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raite d'un aspect important des enquétes qui est trop souvent 
neglige ou pas suffisamment mis en lumiere: la valeur des operations entre- 
orises pour réaliser une enquete est étroitement liee a etablissement du 
olan de cette enquéte. Au cours des dernieres decades, | éventail des 
methodes de sondage probabiliste s'est etendu de facon considerable en- 
trainant un accroissement des possibilites des chercheurs et des statistici- 
ens, pour découvrir des faits et fournir des informations adéquates touchant 


a un grand nombre de sujets relatifs aux phénomeénes sociaux, demograph- 
Mais, ces possibilites sont effectives seulement 


Cet article t 


‘ques et economiques. , tne 
quand la conduite de |’enquéte conforme au olan qui en avait ete eiabli. 


La concordance entre |'exécution et le plan de l'enquéte ne peut se 
supposer et naturellement sera realisee seulement SI des actions y tendant 
sont entreprises. 

L'auteur propose des méthodes pour établir des systemes de controle et 
des techniques permettant d’assurer la surveillance des operations et ae la. 
la fidélité jugée indispensable; on attache une attention toute particuliere 
a l'appréciation du plan, au contrdle du recueil des donees, a la fagon de 
ies transcrire et de les exploiter. a | élaboration des estimations et enfin, a la 
preparation du rapport final. On donne un grand eventail de procedes 
théoriques et aussi de méthodes empiriques. Toutes ces methodes ne doi- 
vent pas étre appliquées systématiquement a toutes les enquétes. Pour une 
enquéte de petite envergure, les modalités de contrdle peuvent étre simples. 
Pour une enquéte plus étendue et plus importante. les mecanismes de con- 
trdles devront étre plus nombreux et plus pousses. Ce manuel a pour objet 
de degager les principaux aspects ce l’enquéte qui devront étre contrdle et 
de proposer les outils de contréle appropriés. Il a aussi pour objet den- 
courager une certaine habilete propre a préevoir les risques d’erreurs 
possibles dans le déroulemeni d'une enquéte et de chercher les moyens 
pour enfermer celles-ci dans des limites raisonnables. 


to our readers V 


“ anos lecteurs 


Operational Control of Sample Surveys treats a Significant aspect of sur- 
vey activity that is too often neglected or underemphasized. This is the 
Steps that must be taken to assure that actual execution of a Survey is 
faithful to its design. The imaginative development of probability sampling 
in recent decades has increased enormously the potential of researchers 
and statisticians to discover facts and produce relevant information on a 
wide range of social, demographic, and economic matters. But that poten- 
tial is realized only when conduct of the survey mirrors the design under 
which it was planned. Concordance between execution and design cannot 
be assumed, and indeed will exist only if positive actions are taken to see 
that it does occur. 

Operational Control offers guidelines for establishing control systems and 


a nuestros lectores > 


Este articulo trata de un aspecto significativo del proceso de las encuestas 
que frecuentemente no se tiene en cuenta 0 es escasamente acentuado. Se 
refiere a los pasos que deben tomarse para asegurar que la ejecucion de una 
encuesta sea el fiel refleio de su diseno. El ingenioso desarrollo del mues- 
treo probabilistico en las Ultimas décadas ha incrementado enormemente el 
potencial de los investigadores y estadisticos para descubrir hechos y 
obtener informacion relevante en el amplio campo de los hechos sociales, 
demograficos y econdmicos. Pero el potencial se materializa solo cuando 
la ejecucion de la encuesta se refleja en el diseno bajo el cual se planea. 
La concordancia entre ejecucion y diseno no puede ser supuesta, y en 
realidad solo existira si se llevan a cabo ciertas acciones dirigidas a obtener 
tal concordancia. 

El autor ofrece guias para establecer sistemas y tecnicas de control para 
mantener vigilancia sobre las operaciones, y por lo tanto, proporciona la 
seguridad nacesaria. Se da atancion a la validez del diseno, al control 
sobre la recolecci6én de informacion, a la edicion y al procesamiento, y a 
la preparacion de los estimadores e informe final. Se presenta una variedad 
considerable de procedimientos formales y de artificios de control subjetivo. 
No todos estos métodos deben ser utilizados en cado encuesta. Para un 
proyecto pequeno, la estructura de control puede ser modesta. Para una 
encuesta mayor y mas importante, los mecanismos de control deben ser 
mas extensos y compresivos. El estudio tiene por objetivo identificar as- 
pectos claves de operacion que deben ser controlados y sugerir herramien- 
tas de control apropiadas; pero aun mas, se propone fomentar un habito de 
pensamiento que reconozca el riesgo de error en cualquier procedimiento 
de encuesta y que busque los medios de mantener tal riesgo dentro de 
limites aceptables. 


techniques for maintaining surveillance over operations, and thus providing 
the needed assurance. Attention is given to validation of the design; to con- 
tro! over data collection, editing, and processing; and to the preparation of 
estimates and the final report. A considerable variety of both formal pro- 
cedures and more subjective control devices is presented. Not every one 
of these methods should be used in every survey. For a minor undertaking, 
the control structure can be modest. For a larger, more important survey, 
the control mechanisms should be more extensive and comprehensive. The 
objective of the manual is to identify key aspects of operation which should 
be controlled, and to suggest appropriate control tools; but more than that, 
to encourage a habit of thinking that recognizes risk of error in any survey 
procedure, and seeks means of keeping that risk within tolerable bounds. 
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Operational Control of Sample Surveys 


Walt R. Simmons 


1. Introduction 


The activities of nations, governments, busi- 
ness enterprises, universities, and individual 
persons are complex, but all undertakings 
directed by rational men can be described in 
Outline as formulation of objectives, planning 
to meet those objectives, and execution of the 
plans. Most administrators are coming to realize 
that wise and productive social action must be 
based on extensive and detailed factual in- 
formation. Increasingly that knowledge is being 
acquired, or at least sought, through the mech- 
anism of the statistical sample survey. By what- 
ever means initial primary objectives may have 
been determined, it is true for the sample sur- 
vey, as for other undertakings, that rational 
behavior calls for planning and then for carrying 
Out the plans. 

The planning activity is responsible for pro- 
ducing the survey design; execution is com- 
prised of a series of activities termed survey 
Operations. As will be made clear in subse- 
quent sections of this manual, Survey design 
and survey operations are (or should be) inter- 
active. Moreover, if the purposes of design are 
to be fulfilled in operations, those operations 
must be in reasonable accord with the design- 
ers’ intentions, just as in building a house 
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actual construction must follow the architect's 
plans. It may be less obvious to the inexperi- 
enced surveyor that resulting correspondence 
between survey design and survey operations 
is very likely to be highly disappointing unless 
positive steps are taken to guarantee that 
correspondence. The purpose of this mono- 
graph is to provide guidance to survey man- 
agers in adopting techniques which will give 
much greater assurance that performance will 
match intention, even though absolute guarantee 
is never certain and can only be approached 
with continuous vigilance. Special emphasis 
will be placed on the merits of independent 
verification and replication, taking advantage of 
the principle that the probability of the same 
error being made by two independent measure- 
ments is the product of probabilities of error for 
the individual measurements and therefore, 
much smaller. 


2. The Principle of Control 


The competent statistician has learned and 
must never forget that he lives and operates in 
the shadow of an axiom: 

The conduct of a survey that lacks built-in 
operational controls will not mirror the design. 
Several corollaries follow immediately. lf the 
survey is not carefully controlled, money and 
time will be wasted, and results will be of 
doubtful and unknown validity. The survey will 
fail to produce solid information, and may 
actually yield misinformation. The objective 
which was hoped for will be lost. 


Why does an uncontrolled survey fail? It may 
not be that planning was weak, or instructions 
unclear, or that enumerators, coders, and clerks 
were unskilled or dishonest — although any of 
these factors and others may be present. 
Rather it is that all of us at times misunderstand, 
make mistakes, encounter unforeseen situa- 
tions, and take actions which we had not earlier 
intended. (Indeed some survey statisticians 
say ruefully, ‘If an unfavorable event can hap- 
pen, it will.) The survey can be successful 
then only when procedures are devised which 
collectively will assure that deviations from the 
intended design are kept within reasonable 
bounds. This is the principle of operational 
control, but how is it secured? That cannot be 
answered in a word, nor in a few sentences. 
This manual will identify the points at which 
controls are most necessary, and offer tech- 
niques and procedures for control over oper- 
ations at those points. The recommended 
methods have been found effective, but the 
most valuable message the manual carries 
is the encouragement it may give the practicing 
Statistician or researcher to develop the habit 
of subjecting a// survey operations to verifying 
or validating procedures to the maximum extent 


that his imagination and resources will allow. — 


Some readers may say, “‘Of course the sur- 
vey should be carried out as planned. Every- 
one knows that. Any good director or super- 
visor will see that his employees do their jobs 
properly.”’ The task is difficult, though. Good 
intentions, and vigorous general supervision 
are not enough. Structural controls are re- 
quired. These are the tools of strong super- 
vision. Skill in their use can change failure into 
SUCCESS. 


3. Types of Survey 


There are many kinds of sample surveys. They 
differ with respect to subject: for example, they 
may treat human or animal populations and 
their characteristics — those which deal with 
people may concentrate on particular aspects 
of the population such as births and deaths, 
health, unemployment, income, hours of work, 
industrial or occupational attachment: or the 
universe of interest may be crops planted and 
harvested; or industrial production and Sales: 
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or housing; or domestic and international trade: 
or a host of other areas. The principal data- 
collecting instruments vary: interview of per- 
sons in their homes, or at their places of work; 
transcription of written records, such as birth 
registrations or payroll or tax lists; In some 
circumstances, mail questionnaires distributed 
and collected through a postal system are 
appropriate. Surveys may be continuing over 
time, or intermittent, or conducted on a single 
occasion. They may be local or nationwide in 
scope. If nationwide, they may be centrally 
directed, or they may comprise integration of 
regional, district, or local efforts. The overall 
plan may be a scientific probability design, or 
it may be an ad hoc haphazard collection of 
data through some convenient administrative 
mechanism. The undertaking may be a major 
enterprise involving much staff, equipment, and 
large sums of money. Or it may be a minor 
effort carried out by a few people, or even a 
single person, in a brief period of time. The 
general purpose may be a legal or administra- 
tive determination, or it may be research simply 
to improve the state of knowledge. 

Thus it is that surveys differ tremendously in 
kind in the dimensions just indicated, and in 
other ways. Appropriate control systems differ 
accordingly. Some elements of control should 
be common to nearly all forms of sample survey 
endeavor. Other control procedures are pecu- 
liar to particular survey procedures. 

For convenience in presentation, to facilitate 
understanding, and in order to offer material 
that is specific and capable of fairly direct 
application, this manual treats control of statis- 
tical operations in the context of one broad 
class of sample surveys. If a thorough grasp is 
acquired of the methods described it is be- 
lieved that knowledge will permit a statistician 
to construct modifications which will be suitable 
for many other classes of survey. 

The type of survey which the reader is asked 
to have in mind as operational control is dis- 
cussed in the present manual, has these gen- 
eral characteristics. 

A. It is a large-scale, continuing or inter- 
mittent survey of a human population and its 
characteristics. Particular emphasis is placed 
on demographic, sociological, and economic 


features, with health and vital events likely re- 
ceiving prominent attention. 

B. The overall plan is a probability design. 
The sample structure may be relatively simple, 
or it may be complex, involving stratification, 
clustering, more than one stage, more than one 
phase, elaborate estimating equations, and 
other refinements. But it must possess one key 
attribute: every unit in the target population 
must have a known non-zero chance for inclu- 
sion in the sample. 

C. The main data collection is handled by 
field agents. At most points in the manual, the 
agent secures his data through interview, but at 
some through transcription of records. 

D. The survey is centrally directed, although 
usually supervision of data collection is exer- 
cised through several district or field offices. 

As the researcher or survey director adapts 
the recommended control procedures to his 
particular circumstances, which likely will differ 
in at least some degree from the model survey 
which is the background of the manual, he 
may do well to proceed in accordance with 
these guidelines: For any phase of the survey 


operation: 

(a) Decide what is desired to be accom- 
plished: 

(6) write instructions for doing it: 

(c) do it; 


(d) devise and execute a checking procedure 
to insure that it is being done correctly and is 
completed in accordance with the instructions. 


4. Outline of the Operational 
Control System 


Subsequent sections of this manual will 
deal with six phases or subsystems of an over- 
all operational control system which is intended 
to assure within the limits of available resources 
that the ultimate survey product will in fact ful- 
fill the original objectives of the undertaking. 

4.1. Reconciliation of Design and Objec- 
tives. This first phase is the least specific of 
the control subsystems. It is at one time the 
most obviously needed, yet often overlooked, 
and possibly most important single validation 
of survey effort. The concept is a simple one. 
It envisages conversion of general purposes 
into specific objectives. It accepts tentatively 


the survey plan which has been prepared. It 
next assumes that the plan is successfully exe- 
Cuted, and looks at the likely hypothetical out- 
put. The critical issues are whether that output 
answers the original questions and satisfies 
the survey purposes. Often for the initial plan, 
the answer will be ‘‘No it will not.’’ Then the 
initial plan must be modified, and the process 
repeated until there is thoughtful agreement 
that if the adopted plan is faithfully executed, 
there is reasonable expectation that the survey 
will accomplish its purpose. 

4.2 Verification of Theoretical Design. This 
manual does not discuss in any detail the 
mechanics of formal sample or survey design. 
These fundamental matters are treated in text- 
books, other manuals, and the technical appen- 
dices of many survey reports. It is assumed 
here that design work in the general sense has 
been done competently. Nevertheless there 
are a number of respects in which the formal 
design needs to be checked and validated. 
One needs only recall the tragic comment of 
the design engineer, witnessing the collapse of 
a new bridge during opening ceremonies: 
“There is that missing decimal point!” 

4.3. Contro! over Data Collection. It is in 
this third phase of control that survey manage- 
ment is most on trial, and where success or 
failure of operations is most likely to be deter- 
mined. A series of control points and processes 
must be established, which collectively promote 
agreement between the character of intended 
evidence and resulting reported data, and which 
minimize the introduction of extraneous, incor- 
rect, and dishonest data. 

4.4 Control of Data Processing. Building 
upon a base of the output of data collection, 
all surveys pass through a phase of converting 
the raw data into edited, coded evidence suit- 
able for tabulation and analytic manipulation. 
These steps are production-type processes, 
similar in some respects to those of industrial 
manufacturing, where quality-control proce- 
dures were first developed. They can be kept 
within acceptable tolerance levels with the 
Classic quality control techniques. But there 
are also special types of checking that are par- 
ticularly suitable for coding operations, and 
Statistical devices for validating a sequence of 
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processing operations. These will be set forth 
in this manual. 

45 Control of Tabulation and Estimation. 
The essence of tabulation of edited and coded 
data is straightforward arithmetic. Even for 
complex sample designs, inflation of sample 
data to universe estimates, and more elaborate 
estimation of population parameters can be 
specified by rigid mathematical formulae. It Is 
reasonable to expect that these processes can 
be carried out with a high degree of accuracy, 
but it is also reasonable to imagine that it is 
possible for there to be mistakes in arithmetic, 
algebra, statement of instructions, program- 
ming (when tabulation is accomplished on an 
electronic computer or other electric or me- 
chanical equipment), and in the treatment of 
Outliers and other unusual items of data. All 
these errors are likely to occur. They can be 
very destructive unless positive provision is 
made for their control. 

4.6 Control over Report Manuscript and 
Publication. \f the survey has been well de- 
signed and a powerful set of operational con- 
trols has been in effect up to this point of the 
Survey, One can expect that valid data have 
been produced, and that skillful analysis will 
lead to a good report. One more control sub- 
system is necessary. This subsystem has two 
aspects. One is simply verification of transcrip- 
tion and arithmetic which convert worksheet 
results to numbers, tables, and charts in the 
manuscript and final report copy. The other is 
some scheme of review or independent analy- 
sis of judgments and conclusions reached by 
the authors of the report. The review should 
include editorial scrutiny for clarity, removal of 
ambiguity, and attractiveness of presentation. 
This final control procedure is again, like the 
first subsystem, somewhat more subjective and 
less specific than the other four subsystems, 
but that does not justify its omission. It would 
be a personal tragedy for the Survey director or 
Statistician to have produced sound and rele- 
vant data through great and careful effort, only 
to have the end product misrepresent those 
findings, or be disregarded because it could 
not be understood, or was simply too unattrac- 
tive to be read. The statistician should remem- 
ber that usually the majority of his clientele will 
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judge his performance — perhaps not fairly — 
not so much by how he planned or conducted 
his survey, but by the report he prepares of that 
survey, its findings, and conclusions. 


5. Reconciliation of Design 
and Objectives 


5.1. General Considerations. On too many 
occasions a survey produces ‘‘beautiful’”’ tables, 
charts and conclusions — and still has little 
relevance or utility for the purposes for which 
the statistical project was undertaken. 

The remedy for these deficiencies must be 
applied early in the project. General purposes 
should be connected to specific objectives. 
Ideally these should be expressed in a series 
of written questions which the survey needs to 
answer. The questions should be quite specific, 
and they should be expressed in defined terms, 
and capable of being answered in measurable 
quantities. Scope of the survey with respect to 
subject matter, definition of population to be 
Surveyed, geographic coverage, time span, 
length of reference period, and frequency of 
enumeration must be clearly stated. Decisions 
are made concerning whether results are need- 
ed for only the total universe as a whole or 
separately for certain geographic subdivisions 
and specified domains within the universe. At 
least general levels of acceptable tolerances or 
Sampling error are set. These steps clarify and 
put into useful operational format initial require- 
ments for the survey. 

Assume that an initial survey plan has been 
outlined and tentatively accepted. Consider 
next the expected output of such a survey. Does 
it match the requirements? The first round of 
planning likely will not have produced a com- 
plete match. The projected survey may be de- 
ficient in some respects, or it may contemplate 
the production of information which is not really 
needed, and thereby be more expensive than 
is justified. Appropriate adjustments in the plan 
are made. Note that the title of this section is 
“Reconciliation of Design and Objectives.” It 
may be that considered study of the situation 
results in the conclusion that one or more of 
the originally stated objectives are not really 
feasible. On balance, the evidence may be 
that certain desired attributes of the population 


are not measurable through any survey tech- 
niques which circumstances permit. In that 
situation, it is the objective which should be 
modified, so that it is re-expressed in terms 
that are measurable. This pattern of review, 
comparison, and modification of plans and ob- 
jectives is repeated until satisfactory agreement 
between objective and plan is obtained. The 
cost of any compromise in objective which may 
have been necessary is more than offset by the 
value of having chosen a plan which can be 
carried out, as contrasted with taking a course 
which is doomed to failure from the very 
beginning. 

5.2 An Illustration. Let us suppose that a 
government has decided it is necessary to 
undertake a study of the human population of 
the Province of West Dakota. Officials say that 
they wish to know how many people there are, 
how the size of that population is changing, the 
health status of the people, and the situation in 
terms of gainful employment. 

Assume that designers have prepared a ten- 
tative plan for this survey. They decided that 
the general vehicle should be a household inter- 
view of approximately 10 per cent of the popu- 
lation, selected through a multi-stage area sam- 
ple of West Dakota. Questionnaires have been 
constructed, and relevant definitions and _ in- 
Structions written. Preliminary plans have been 
set for securing necessary resources in per- 
sonnel, space, equipment, sampling materials, 
publication facilities, and other supporting com- 
modities. A first formal sample design has 
been developed. Drafts have been completed 
of instructions for all phases of the survey, in- 
cluding data collection, processing and tabu- 
lation. In short, a comprehensive and compe- 
tent job of early planning has been done. 

In the first phase of operational control, the 
assumption is made that all production will be 
Carried out as planned. The question is, will 
the survey then satisfy its objective? This in 
turn raises such questions as the following, 
each of which should be exnlored and resolved 
before the survey goes into ':@ field and even 
before much energy is devoted to planning the 
more detailed and specific control procedures 
to be applied to other and later phases of the 
Survey. 


A. \s the geographic scope of the survey 
and of West Dakota clearly and unambiguously 
defined? If boundaries are in any respect un- 
certain, they must be clarified. In this respect, 
and in all subsequent checks, a positive effort 
should be made to bring final definitions and 
determinations into as close accord as feasible 
with the desires of the sponsors of the survey 
— remembering always that realities may dic- 
tate modifications of original preferences, to 
the end that the task attempted is a feasible 
one. 

B. Do the definitions provide for whether 
every person who may be encountered in the 
Survey iS properly included or excluded from 
the universe of study? Are there classes of 
persons who belong in the universe but are 
likely to be missed unless special provision is 
made for them? Is there a firm decision on 
whether a de facto or a de jure concept of pop- 
ulation has been adopted? Is the effective date 
of the count specified in an acceptable fashion? 

C. Since ‘“‘household’”’ is one of the build- 
ing blocks of the data collection procedure, and 
since it is a component of the definition of per- 
sons to be included in the sample, is the defi- 
nition of the term consistent with the intended 
inclusion of persons in the population? For ex- 
ample, if it is intended that West Dakotans who 
have not lived in the Province for a year be 
excluded, do household membership rules also 
exclude such persons? 

D. The questionnaire should be reviewed to 
see if it meets the implications of the following 
list of questions and the principles which they 
reflect: (1) Will information be provided on 
all required subjects? (2) Does the question- 
naire ask for the right information, in terms that 
are properly defined, that can be readily under- 
stood by respondents, and in sufficient detail 
for appropriate analysis? (3) Will the result- 
ing data and classification systems yield in- 
formation not alone for the total universe, but 
for desired geographic and demographic sub- 
universes? And (4) Will the survey or known 
other sources provide necessary and appro- 
priate denominators for any rates or ratios 
which analysts desire? Items E, F, G, and H 
immediately below offer specific illustrations of 
how some of these tests can be applied. 
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E. The questionnaire provides for recording 
head count, sex, age, occupation, employment 
Status, and health status for each person In 
every sample household. Passing over, for the 
moment, the need for data on events as distin- 
guished from data on current condition or 
Status, do these questions provide all the ma- 
terial needed for meeting the survey mission? 
IN answering this question, it is necessary to 
go beyond the words used originally by the 
sponsors and to try to satisfy their intention. 
Sex and age were not explicitly requested by 
the sponsors, but they clearly are necessary to 
any useful analysis of survey data. The review 
probably would conclude that relation to head 
of household also is needed, and should be 
added to the questionnaire. 

F. Sponsors asked for the situation in terms 
of gainful employment and health status of the 
population. Do employment status and health 
Status in the questionnaire, along with the other 
items provide the answers? Certainly alone, 
just by themselves, two such questions would 
not. Consider first employment status. An ap- 
propriate course will depend on the nature of 
the economy in West Dakota, but very likely it 
will be necessary for the inquiry to be a good 
bit more specific — perhaps along the follow- 
ing lines. In addition to formal occupation, 
there should be questions that will distinguish 
among housewife, small children, children in 
schools, and persons too old or unable to work, 
as well as determination of whether the person 
is employed or unemployed. Employed is itself 
capable of a variety of definitions, and the one 
cnosen should match the needs of the spon- 
sors. It may well be that knowledge is desired 
too of class of worker, i.e., whether a person is 
self-employed, an employee of a government, 
or working for a private employer. Is the ob- 
jective to secure employment status, now, dur- 
ing the past week, usually, or over some other 
time period? 

G. Health Status is an even more complex 
concept than employment. Probably no set of 
questions administered by interview can yield 
a fully satisfactory picture of the health status 
of a person or population. This is almost cer- 
tainly an area of the survey in which the Original 
charge will need be modified in order to estab- 
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lish an objective that can be attained. Quite 
possibly the target may be revised to answer 
the following questions: (1) Is the person 
physically impaired, 1.e., blind, deaf, missing 
arms or legs, unable to feed himself, or to move 
about? (2) Is he able to do the type of work 
to which he is accustomed, or which persons of 
his age and sex can usually do? (3) Does he 
have any of a specific list of chronic diseases? 
(4) How many days in the past month has he 
been confined to bed all or most of the day? 
These questions will not produce a complete 
description of the health status of the popula- 
tion. But they, or other similar questions, sup- 
ported by data on deaths and causes of death 
(to be discussed shortly), can yield valuable 
evidence on health for the government officials. 
Since they represent inquiries and objectives 
that are attainable, they promise less dis- 
appointment than would a more amorphous at- 
tempt to enumerate health status. 

H. The survey is asked to provide informa- 
tion on changes in the size of the population. 
Presumptively, the survey planners will have 
converted this quite general request into spe- 
cifics. Assume they have decided to mount a 
continuing undertaking, and to repeat the sur- 
vey at annual intervals. Thus once each year 
they will enumerate a cross section of the pop- 
ulation, and this will provide the basis for an- 
nual assessment of net change of the profile 
by age, sex, and other measured character- 
istics. Since health and the dynamics of pop- 
ulation change are also of interest, they have 
provided for enumerating births and deaths. 
(Experience has shown that determination of 
birth and death rates by an intermittent inter- 
view procedure is difficult. But remember, we 
assume in this section that the chosen plan 
can be executed.) Review of this part of the 
plan must assure that procedures are designed 
to count a// births and deaths which occurred 
to persons associated with the Sample areas 
during the reference period (perhaps the year 
Preceding interview). Several valid variants of 
this formula are acceptable. The point is that 
definitions and counting must provide for con- 
sistent formulation of numerators with denomi- 
Nators of the birth and death rates, while atten- 
tion is given also to preventing omission of 


certain classes of persons or events from either 
numerator or denominator, or to erroneous in- 
clusions of persons or events. Particular atten- 
tion must be given to rules for attaching a per- 
son uniquely to an area, and to treatment of 
migration into and out of West Dakota during 
the reference period. Since the interviews are 
with the living population, special attention 
must be given to proper inclusion of deaths, 
and most especially in instances in which death 
may have dissolved a household. 

lt is again emphasized that it is not the pur- 
pose of this manual to recommend ideal survey 
designs, but rather to offer guides for controlling 
surveys after they have been planned. There is 
an extensive literature on designs for measur- 
ing population change and vital events. See 
for example references [10,11,12,13]. 

5.3. Confirmation of Plan. Having reviewed 
each of the main features of the survey plan in 
terms of specific objectives, and likely having 
modified the original requests in some respects, 
it is desirable to secure the concurrence and 
approval of sponsors, i.e., the officials who have 
authorized or commissioned the undertaking, 
before proceeding further. The statistician or 
project director may be hesitant to face the 
sponsor at this stage, having chosen a plan 
which may offer somewhat less than the spon- 
sor expected, and not yet being absolutely 
certain that he can ultimately produce even the 
lesser product. But it is far better to make cer- 
tain at this point that a fully agreed upon task 
is being undertaken than to discover at the end 
of the survey that what he has done is un- 
acceptable. Perhaps any differences of opinion 
at the early stage can be happily resolved with 
modest effort. It is quite likely that the spon- 
sors will be pleased to see the researcher 
proceeding in an orderly, business-like manner, 
and to learn in fairly specific and concrete form 
just what kind of evidence and information the 
survey is likely to produce. 

5.4 A General Reminder. From time to 
time in this manual, for purposes of illustration 
and in order to focus on a specific point, there 
will be discussions of a particular example, as 
in section 5.2 above. It should be remembered 
that these are examples only. Surveys and sur- 
vey objectives differ widely. A particular pro- 


cedure which is appropriate, or even neces- 
sary for one survey, may be ill-suited to an- 
other. The user of the manual is urged then 
not only to become acquainted with the specific 
procedures which are suggested, but even 
more importantly to seek understanding of 
principles behind the procedures and of the 
fundamental purposes of those procedures. For 
instance, in this first phase of operational con- 
trol, discussed in section 5, the central purpose 
is to take positive steps to assure that the 
chosen strategy for a survey, if successfully 
executed, will in fact satisfy the objectives of 
sponsors. It means that the operating statisti- 
cian starts out with the confidence that if he 
does his job well, he will be engaged in a pro- 
ductive enterprise. He avoids the risk that even 
if he should do a good technical job, the re- 
sulting data might be unsatisfactory, or even 
serve no useful purpose, and might be of no 
interest to any significant consumer. 


6. Validation of Theoretical Design 


6.1 Scope of the Second Phase of Control. 
From this point on it is assumed that a general 
Strategy has been agreed upon, following 
thoughtful review, and final confirmation by 
sponsors and administrators as well as by the 
project manager. Indeed a reasonably com- 
plete design for the survey has been laid out. 
At this point, a second control phase should be 
introduced before data collecting operations 
begin. This control is technical in nature, in 
the sense that it is intended to validate the 
work of the mathematicians and Survey statis- 
ticians who are the professional survey design- 
ers and planners. 

It should be made particularly clear at this 
point that the introduction of checks and vali- 
dating procedures for the chosen design is not 
in any sense a reflection on the wisdom or skill 
of the designing statistician. Rather it is recog- 
nition of our sound guiding principles that the 
risk of error is always present, but can be mini- 
mized through replicating and control pro- 
cesses. The true professional statistician will 
welcome and even insist that his design work 
be validated. 

6.2 Rationale of Design. Remember an 
assumption of this manual is that a generally 
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sound overall design has been formulated. It 
has now been accepted. The control objec- 
tive in this phase is not to search anew for the 
best possible design, but rather to review the 
chosen design, to look for flaws in details, per- 
haps to introduce modifying improvements, in 
order to augment confidence that the blueprint 
for the survey does indeed offer a workable 
pattern, and thus to confirm that the design can 
reasonably be expected to be successfully 
executed. 

If circumstances permit, the ideal course Is 
to obtain an independent review of the total 
plan by one or more competent Statisticians 
who are knowledgeable of the survey objec- 
tives, but who were not the responsible original 
designers. The principle which directs this 
course is that two or more largely independent 
judgments are less likely to make the same 
mistakes, or overlook the same shortcomings. 
If independent reviews are not available, the 
task is more difficult, and more dangerous, but 
valuable review is still possible. The same 
statistician on a second trial can uncover weak- 
nesses in an earlier effort if he consciously 
approaches his task with a critical and ques- 
tioning frame of mind. He can deliberately con- 
struct alternatives, compare the consequences, 
and choose the better route. Often he can 
arrange for associates or subordinates to check 
particular aspects of the design. 

One fruitful technique is to consider the im- 
pact which unexpected circumstances or pop- 
ulation parameters would have if they should 
be encountered in conduct of the survey. For 
example, the design may have been predicated 
on speculation that the population coefficient 
of variability of a key item is 40 per cent within 
primary sampling units (P.S.U’s) and 10 per 
cent between P.S.U’s. Construction of a table 
of sampling errors which are functions of within- 
and between-variances over considerable 
ranges from the speculated values will identify 
areas of allocation of sampling resources to 
within- and between-P.S.U’s in which the de- 
sign is most stable or robust. This exercise may 
confirm the original allocations or may suggest 
a modification that promises better insurance 
against the unexpected. 
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In some instances the review may suggest 
fortifying the initial design, by introducing an 
element of flexibility into the sampling or esti- 
mating plan. For example, scarcity of data 
may have led to an uncertain determination of 
sample size or of unit costs. It may be feasible 
to break the parent sample into, say, four ran- 
dom subsamples or panels, and then schedule 
operations so that preliminary evidence from 
the first panel can be used to decide whether 
three or all four panels must be used to secure 
needed precision or to stay within available 
budget. Or perhaps the estimating scheme 
specified the use of a ratio estimate, expecting 
to take advantage of correlation between two 
variables, x and y. It may be wise to modify 
the plan so that the ultimate decision on type 
of estimate will depend upon evidence of the 
degree of correlation which materializes in the 
actual survey data. 

Suppose the survey seeks to measure the 
birth rate by collecting interviews in a sample 
of land segments. The estimating scheme 
makes certain assumptions about the charac- 
teristics of persons who migrate into and out 
of those segments during the reference period. 
These assumptions are critical to final results 
and should be reviewed very carefully. With 
only modest modification of field procedures 
and the addition of a few questions in the inter- 
views, It often will be possible to build into the 
survey techniques for evaluating the migration 
assumption, and possibly strengthening the 
estimates of birth rate. 

We do not treat in any detail at this point 
the issues of relevance, specificity, clarity, and 
answerability in the questionnaire, assuming 
that they either have been covered in phase | 
control, or will be resolved in pretests and pilot 
Studies to be treated subsequently. But the re- 
viewer must necessarily keep in mind that the 
real survey, as distinct from a model on the 
drafting board, will be dealing with responses 
in terms of numbers, replies of ‘‘yes’’ or ‘‘no”’, 
names of people and places, and classifying 
attributes rather than the more tractable x’s and 
y's of his equations. Accordingly, he should 
try to assure himself that there will be congru- 
ence between the collected data and the math- 


ematical models developed in the planning 
phase. 

The design should be checked for presence 
and reasonableness of a number of provisions. 
some of these are discussed in subsequent 
sections. A supplementary five-point check list 
is noted at this point. 

A. Is there an adequate plan for handling 
non-response? First, there need be standards 
set for maximum allowable non-response, both 
in failure to respond by the entire sampling 
unit, and in omission of some items of data for 
sampling units which report incompletely. Sec- 
ond, a continuing surveillance procedure should 
be established, which, along with corrective 
measures, will assure that the standards are met. 

B. It must be expected that both environ- 
mental circumstances and the economics of 
Operation are such that not all non-response 
can be prevented. Therefore, procedures must 
be established for imputing values for the miss- 
ing information into survey averages and totals. 
Remember that imputation, either explicit or 
implicit, cannot be avoided. It is not a question 
of whether to impute for missing data, but only 
how to impute. 

C. Is the contemplated computational pro- 
cedure arithmetically sound? Perhaps not all 
possible contingencies can be foreseen at this 
stage, but one can check to see if plans pro- 
vide for carrying appropriate numbers of sig- 
nificant digits, recording of needed sub-totals, 
specifying the handling of blanks and ‘‘impos- 
sible’ items of input, avoiding steps which 
might involve division by zero, and making sure 
that the size of expected sums and products is 
within the capacity of chosen computational 
equipment. 

D. Will the processing and tabulating 
scheme provide an output record of statistics on 
important editing, imputation, and adjustment 
steps? For example, the bookkeeping should 
provide a record of the number of unit non- 
responses, numbers of missing items, the vol- 
ume of key imputations and the sizes of ratio 
adjustments. This record is invaluable to later 
evaluation of output. It can save the statistician 
from drawing unjustified conclusions. If the 
record showed that it had been necessary to 
impute, say, 60 per cent of the data for a par- 


ticular item, the statistician would know that 
the survey had produced no reliable data at 
all on that item. 

E. Does the plan provide a comprehensive 
and detailed blueprint of document flow and 
the transmission of information (data) through- 
out all phases of operation? The working path- 
way from respondent to publication should be 
Clearly delineated. This matter is discussed 
further in section 8.2. 

F. Is the total procedure adequately docu- 
mented? There should exist at least one com- 
plete ‘‘brass-bound file’ or master account, 
which describes in full detail all aspects of the 
design and associated procedures. This is 
necessary to protect against faulty memory on 
the part of designers, as well as against the 
possible death or absence of those designers 
at a later stage, when it will often be necessary 
to know precisely what the plan of the survey 
or its operation was on a critical point. It also 
is the foundation for preparation of instruction 
manuals, which should be written for all phases 
of the survey operation, and it is the authority 
of last resort for the analysts who finally draw 
conclusions and publish the survey results. 

6.3. Mathematical Validation. \t is recom- 
mended that a second statistician check the 
mathematical and statistical logic, and the alge- 
bra of the entire theoretical formal design. Again 
it is observed that such a step is no reflection 
on the ability of the principal designers, but 
only a reasonable precaution. All of us make 
mistakes. Even the most skillful and experi- 
enced person will occasionally forget to square 
an expression, will use a variance where he 
should have used a rel-variance, neglect a re- 
mainder in a series expansion when the remain- 
der is not trivial, mishandle the algebra in cal- 
culating an expected value, use an improper 
variance formulae, or err in many other ways. 
lf the error is undetected, the entire survey 
structure may rest on a foundation which con- 
tains a serious or even fatal flaw. 

Often, in addition to straightforward depen- 
dent verification of the original mathematics, it 
is both desirable and feasible to secure a par- 
tially independent derivation, proceeding from 
slightly different assumptions and arriving at 
approximately comparable conclusions. When 
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this is possible, it provides the best kind of 
insurance. | 

6.4 Verification of Arithmetic. This step !s 
an adjunct of the one just described, is taken 
for the same reasons, and is even more ob- 
viously necessary. All arithmetic and numerical 
calculations in the designing and planning 
phases must be verified on a 100 per cent 
basis. Several operational hints may be helpful: 

A. Particular attention should be focused 
on location of all decimal points. 

B. The scale of all results, both intermedi- 
ate and final, should be considered for reason- 
ableness. Often a supplementary approximat- 
ing calculation using only two or three signifi- 
cant digits and powers of 10 will produce a 
confirming figure for comparison. 

C. Often, for a complex plan — perhaps a 
two-stage, stratified, and post-stratified design 
— it is useful to make parallel computations for 
a simpler design, and to compare the two re- 
sults. The tactic here is of course to seek con- 
firmation of a plausible relationship between 
results from the extended calculations for the 
complex design and those from the simpler 
arithmetic where there is less chance for error. 

D. Some of the input for these calculations 
is likely to be speculative in character, and al- 
most certainly to have been transcribed from 
outside and possibly alternative sources. A 
careful and questioning review of the input num- 
bers is warranted. 

E. Perhaps it is looking for trouble to sug- 
gest the possibility that a desk calculator used 
in making calculations may not be functioning 
properly. Yet this can happen, and of course 
One should be reasonably sure that it does not. 
It is easier to believe in possible equipment 
malfunction, but harder to avoid, if part of the 
Planning computations are carried out on 
punchcard or electronic computer machines. In 
such instances, error can have its source in 
either the hardware or in its programming. When 
this type of calculation is involved, it is particu- 
larly important to seek confirmation of key re- 
Sults through simpler approximating techniques. 

6.5 Mesh of Data and Design. Attention 
was Called earlier in the discussion of review 
of the overall rationale of the design, to the 
need for correspondence between collected 
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data and tabulating plans. As the details of 
tabulation and estimation are finalized, a firm 
matrix must be established which translates un- 
equivocally each input datum into the appro- 
priate variable or constant of the design pattern. 
This matrix is a two-way guide. It pictures as 
well the source of data for each variable or 
constant — each x, y, a, b, c—of the formal 
design. An affirmative answer is required for 
every question: ‘Does the plan assure that 
needed data will be forthcoming for this factor 
in the design and tabulation plan?” 

6.6 Mean Square Error. One wishes it 
could be said that every survey plan must in- 
clude ascheme for determining the mean square 
error of the principal survey results. Mean 
square error (MSE) is defined as the sum of 
the variance and the square of the bias of a 
Statistic. The bias is defined as the difference 
between the expected value of the statistic and 
the true value of the parameter which the sta- 
tistic estimates. Variance is the expected value 
of the square of the difference between the ob- 
served statistic and the expected value of that 
Statistic. Total variance includes both sampling 
variance and measurement variance, but does 
not include any bias, whether it may arise from 
sampling or measurement. 

Unfortunately the state of the art does not 
permit the promulgation of a requirement that 
MSE must be specified. But the art and good 
Practice do justify a demand that the survey 
plan provide for means of discovering the 
sampling variance of key statistics, and that 
the statistician take such steps as he is able to 
Supply evidence on measurement variance and 
on bias. No plan is complete without such 
provision. 

Other manuals and sampling textbooks, as 
well as a fairly extensive list of articles in pro- 
fessional journals offer methodology for secur- 
ing sampling variances and for approaching 
evaluation of mean square error. This manual 
offers no recommendations on how the evalua- 
tions are attempted, but does insist that the 
Survey plan deal with the topic, and as a mini- 
mum provide for determination of sampling 
variance. It also suggests the desirability of 
the following provision which is not often found 
in textbooks. We refer to the following situa- 


tion. Frequently the calculation of a precise 
sampling error is an involved computation which 
is expensive, time-consuming, and given a low 
priority in operational demand on personnel 
and equipment. Too often, then, other tabula- 
tions are completed, analysis performed, and 
even published before the sampling errors have 
been computed. To protect against this con- 
tingency, it may be wise for the survey plan to 
include a back-up procedure which is a less 
costly approximating technique for estimating 
sampling variance. That weaker, but still use- 
ful, technique can be utilized in the event that 
the more fully adequate process does in fact 
prove to be either unattainable or late in coming 
forth. 

6.7 Resources and Logistics. Every survey 
plan is predicated upon, or assumes the exis- 
tence of, the wherewithal to carry out the plan. 
This means availability of financial resources, 
manpower of several kinds, space, equipment, 
materials, and supplies, along with an adminis- 
trative and operational structure with the ca- 
pacity to deliver proper and timely logistical 
support at all phases of the survey. 

Examination of the provision for these mat- 
ters calls for the talents of both the statistician 
and administrative aides who are knowledge- 
able of relevant personnel rules, procurement 
channels and regulations, budgetary proce- 
dures, and other management paths. In these 
areas as in others, the specifics vary with cir- 
cumstances, but typically the statistician should 
review the specifications for numbers and kinds 
of personnel required, and check over the avail- 
ability of needed equipment and materials, in- 
cluding especially such items as lists of units 
to be sampled, maps, classification systems — 
both the systems themselves, and the related 
coding manuals — transportation and commu- 
nication devices, file cabinets, duplicating ma- 
chines, transcribing and tabulating machinery, 
typewriters, and the long list of small supplies 
such as paper, pencils, staplers, envelopes, 
punchcards, index tabs, and file folders. He 
will assure himself that the survey will in fact 
have access to any auxiliary data—for in- 
stance, census files, or tax and licensing rec- 
ords — which have been incorporated into the 
plan. 


In validating the manpower and material re- 
quirements, the reviewer is advised to look 
not only at global figures, but to go through unit 
Calculations as well. For example, if the plan 
calls for 100 interviewers, he will wish to think 
about whether and how 100 suitable interview- 
€rs can be found, but also whether 100 is a 
reasonable number. The unit audit approach 
makes such judgments as what is the average 
time for a single interview, the travel time be- 
tween interviews, proper hours of work per 
day per interviewer, days of work per week or 
month per interviewer, likely turnover rate of 
interviewers, i.e., how many interviewers must 
be replaced per month. He will wish to weigh 
these estimates against the total number of 
interviews to be made, and the time allowed 
for the task. The administrative aide will be 
concentrating on the supply side of the markets 
for space, equipment, and materials, and on 
procurement procedures. 

The planned administrative, supervisory, and 
operational structure merits inspection from the 
point of view of sound and efficient manage- 
ment. Do the central organization, the regional 
field supervision, and the local operating ar- 
rangements fit the plan? Is the chain of com- 
mand clear-cut? Is there a system of progress 
reporting? Are there pathways for resolving 
unforeseen problems in a manner that does 
minimum violence to the overall plan? Is the 
survey director in a position to exercise real 
administrative control over the entire operation? 

The proposed timetable for conduct of the 
survey should be given a new scrutiny in the 
light of all other checking that has taken place. 
The examination will seek to confirm that it both 
meets requirements of the survey sponsors, and 
passes critical challenges as to feasibility. 

Hopefully the overall review of availability of 
resources and of logistical arrangements finds 
the plan sound in these respects. It it does, 
fine. If it does not, then a further reconciliation 
is called for, this time to bring the newly vali- 
dated design into agreement with a possibly 
modified set of objectives. Just as was true at 
the conclusion of the initial series of reviews 
discussed in section 5 above, any painful as- 
pects of this reconciliation will be less severe 
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at this point in the survey than they would be if 
postponed to a later stage. 


7. Control Over Data Collection 


7.1 Planning and Operations. Once upon 
a time, a complaining cockroach who felt badly 
mistreated sought advice from a bedbug, who 
seemed to the cockroach to live a much more 
comfortable and pleasant life. After a little con- 
sideration, the bedbug responded, ‘Yes, | can 
help. My advice to you is to become a bedbug 
like me.’’ The cockroach said, ‘‘Fine. How do 
| do it?’’ The bedbug replied, ‘That's your 
problem. That is operations. I’m concerned 
with planning!’ Thus far in this manual we 
have discussed planning, and tried to point the 
way to validating a chosen plan. That is an 
important part of overall operational control of 
the survey, but as was noted earlier, it is in the 
production stages and especially in the actual 
collection of data that managerial control is 
tested most severely, for it is here that inten- 
tions are most likely to be violated, and so we 
turn to some of the steps which can be taken 
to reduce the risk that faulty collection of data 
will impair or destroy the survey. The essence 
of the approach now is not to try for perfection, 
but to attempt to insure that the data will be 
collected in reasonable conformity with the vali- 
dated, chosen plan. We are no longer saying 
how the job should be done, but only trying to 
guarantee that it is done as planned. 

Section 7 presents and discusses control 
mechanisms for two separate phases of data 
collection. Sections 7.2, 7.3, 7.4 and 7.5 are 
concerned with a group of control and quality- 
enhancing functions which are key elements of 
the data collecting process, but which precede 
the main body of field enumeration. Section 7.7 
on sampling controls proposes devices of which 
some are pre-enumeration controls, and some 
are concurrent with field enumeration. The re- 
mainder of the chapter is devoted to controls 
over the major interviewing or enumerative Op- 
eration itself. 

7.2 Pre-Tests. Section 5.2 introduced in 
broad outline an illustration in which an house- 
hold interview survey of about 10 per cent of 
the population of the Province of West Dakota 
was to be conducted. The Survey would deter- 
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mine population size, change in population, 
health status, and work status of the people. 
Since specific operational control procedures 
should be adavied to the particular survey, and 
in order to talk in concrete terms, we shall have 
a survey of that type in mind as we describe 
control processes which should be a part of 
data collection. 

Planning will have produced a draft question- 
naire and a preliminary set of instructions for 
interviewing, and for transcription and collec- 
tion of responses. It is not unlikely that two or 
more different versions of the questionnaires 
will have been drafted. It is almost certain that 
a single first draft will not be a perfect instru- 
ment in every respect. Whether there is a 
single first draft, or several, there needs to be 
field trials to discover ambiguities, omissions, 
misunderstandings, irritations to respondents, 
awkward sequences of queries, and other weak- 
nesses in the original drafts. Typically these 
pretests can and should be small-scale trials. 
Often they can be sequential in nature. The 
interviewing should be done by experienced 
personnel at this stage — perhaps supervisors, 
or enumerators who have worked earlier in 
similar Surveys. The persons interviewed should 
be members of the same universe as the sam- 
ple cases, but need not be chosen randomly. 
In fact, it will be better to select subjectively 
a small sample which will reflect what are 
guessed to be extremes in the population, in 
order to get a wider range of test situations. 

A “small’’ sample is recommended. No uni- 
versal specification can be given of just what 
size is best. This will vary with the complexity 
of the questionnaire, the variety of types of re- 
spondents, and the skill with which planning 
has been done. Something like the following 
may fit the situation. 

A. Atleast three different interviewers should 
participate. 

B. Ina first step each interviewer does one 
day's interviewing: perhaps 8 to 12 households. 

C. The interviewers assemble with super- 
visors and the survey managers to compare 
notes, discuss problems, decide on any changes 
that are indicated for the questions, the manner 
of their being asked, or recorded. 


D. A second day's interviewing, similar to 
that of the first day takes place. 

E. Step C is repeated for the second day 
of interviewing. 

F. A third day of interviewing — and per- 
haps a fourth day — followed by further dis- 
cussion and possibly further decisions or 
change may complete the pretesting and result 
in firming up a semi-final questionnaire and 
collection procedure. 

NOTE: The survey manager may wish to do 
some of the pre-test interviewing himself in or- 
der to get a first-hand impression of the climate 
of the operation and leads on some of the pos- 
sible problems. This is particularly likely in 
the smaller scale survey, where less elaborately 
organized pre-testing is warranted. 

In the pretesting just outlined, the partici- 
pants must be alert for a wide variety of pos- 
sible difficulties and other factors which might 
endanger (or promote) success in collecting 
needed data. The following list is in no sense 
exhaustive, but should be suggestive of ele- 
ments to be observed. 

(1) Were there problems in finding the des- 
ignated sample cases? Were the instructions 
adequate for identifying and delineating the 
sample household? 

(2) Was the interaction between the inter- 
viewer and respondent satisfactory, or could 
something be done to improve the contact? 
Was the respondent puzzled, or frightened, or 
angry at being questioned? Did the respondent 
appear to cooperate and try to respond as re- 
quested? 

(3) From the interviewer's point of view did 
the interview move smoothly? Were the ques- 
tions easy to ask, did they have the right se- 
quence? Was the interviewer embarrassed or 
uncertain how to proceed at any stage? Did 
he feel that he had succeeded in asking the 
questions in the manner which had been hoped 
for? 

(4) Were there questions which the respon- 
dent seemed to fail to understand? 

(5) Were there questions which the respon- 
dent failed to answer because he did not pos- 
sess the necessary information? 

(6) Were there questions which the inter- 
viewer suspected were answered incorrectly, Or 


not at all, because the respondent did not wish 
to tell the truth? 

(7) Were there situations encountered in 
which the question, even though answered cor- 
rectly in the literal sense, failed to elicit the de- 
sired information? For example, the question- 
naire might have asked, ‘‘Were you unable to 
work because of illness?’’ An answer of ‘‘yes”’ 
might have been given by a well person be- 
cause he had to take care of children who were 
ill. 

(8) Were all the time references properly 
interpreted? Were distinctions clear cut among 
such points and intervals as “now,” “usually,” 
“‘ever,”’ “past week,” ‘‘three months,” “year 
ago,” “1970"? 

(9) Were there difficulties in establishing 
proper relationships between respondents and 
subjects? Do the instructions seem to be well 
chosen in specifying which sample persons 
should respond for themselves, and for which 
another person should respond? 

(10) In replies to different questions was 
there any internal evidence of inconsistency or 
confirmation? (A well designed questionnaire 
often will provide opportunities for this type of 
check. For example, through comparison of 
age and year of birth, or in listing of household 
membership and noting births of babies, or in 
identifying accidents and loss of time from 
work.) 

(11) Did time-of-day of the interview seem 
to have an impact on results? How did it affect 
willingness or ability to respond? Is some 
period of the day to be avoided or preferred? 

(12) Was there any evidence of better or 
poorer response if other members of a house- 
hold were present when questions were being 
asked of a particular sample person? 

(13) How convenient or inconvenient were 
the physical features of the questionnaire and 
the forms for transcribing replies? Was spacing 
and question arrangement easy to follow and 
read? Was typesize appropriate for lighting 
conditions encountered? Was paper of good 
substance and handy size? Was space ade- 
quate for recording responses? 

(14) For purposes of subsequent schedul- 
ing, a record of times required for interview, for 
travel to and between interviews, and for other 
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tasks should be maintained during the pretest. 
This will be of great value in determining final 
requirements for numbers of personnel, sizes 
of assignments, and reasonable quotas for 
assessing performance. 

7.3. Public Relations. Certainly good work 
in the planning phase will have recognized the 
importance of establishing a favorable climate 
of attitudes in which the survey is to be con- 
ducted and will have made provisions for pro- 
moting that climate. Pretesting may have un- 
covered further facets of public relations that 
deserve attention. “‘Public Relations’ as used 
here reaches over a broad spectrum. It covers 
the areas of legal or official authorization, rela- 
tionships with associated or affiliated or pos- 
sibly competitive organizations, respondents, 
prospective consumers of survey results, the 
communications media, and the general popu- 
lation. Any step that can be taken to secure 
Support of the survey by an affected group, or 
to prevent or neutralize opposition increases 
the chances for a successful venture. The con- 
duct of a survey that is popular is likely to be a 
pleasant operation. Execution in an unfriendly 
atmosphere is difficult and in some circum- 
Stances nearly impossible. To secure the favor- 
able climate may well be worth substantial ex- 
penditure of even limited resources. Attention 
should be given not only to what is done in 
this area, but also to when. 

7.4 Recruitment and Training of Agents. 
One of the truisms of production is “Any job is 
only as good as the people who do it.”’. Plan- 
ning and pretesting have determined how many 
of what kinds of personnel are needed. Person- 
nel management then encompasses setting 
specifications for the various positions, recruit- 
ing, hiring, training, testing, final selection, and 
supervision. 

We assume that adequate recruitment has 
been accomplished. Training will require ori- 
entation of collection agents — and especially 
of the field supervisors and interviewers — with 
respect to the purposes of the survey, and an 
understanding of the policies and perspectives 
of the survey managers. It is recommended 
that collection personnel be given some pro- 
motional materials, and at least one inspira- 
tional lecture by a prominent person, or in any 
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event by a person who has the respect of the 
trainees, as a part of the effort to cause these 
people to ‘‘believe’’ in the survey, and to be 
motivated to do an excellent job. Experience 
has shown that the motivated worker who is 
trying his best will accomplish more than the 
person who may have more inherent ability, 
but who lacks interest or conviction in per- 
forming the task. 

But spirit and good morale are not enough. 
The interviewer, coder, clerk, and supervisor 
must learn the specifics of their assignments, 
too. They cannot do a Satisfactory job until 
they Know what to do and how to do it. First, 
then, comes formal training in the content of 
the instruction manuals. This is followed by 
testing knowledge of that content, both through 
written exercises and examinations and through 
trial performance. The testing is geared to 
identifying satisfactory performance, need for 
further training, and dismissal of those em- 
ployees who show insufficient promise. 

The initial training may include lectures, 
demonstrations by supervisors or leader-agents, 
question-and-answer exercises (without grading 
at first, in order to minimize fear of failure, and 
to develop confidence as the trainee finds he 
can do his task). Small groups of trainees 
working by themselves, quizzing and criticizing 
one another can be helpful. 

Performance testing of interviewers should 
be done under close supervision and observ- 
ance by supervisors. Evaluation is facilitated 
through the use of a check-list observation re- 
port by the supervisors. 

Performance testing of coders, transcribers, 
and other clerks is easier and can be accom- 
plished with structural exercises and during 
processing. These exercises will have known 
correct answers, and thus permit the calcula- 
tion of error rates that in turn are definitive 
measures of performance. 

Amounts of training and testing will vary 
with the initial ability of trainees, complexity of 
the tasks, adequacy of training materials, and 
the tolerance requirements. As for other com- 
ponents of statistical operations, data collec- 
tion and processing performance is subject to 
hazards of both variance and bias. While it 
will not be possible to perform a fully desirable 


arithmetic analysis of the problem, it will be 
useful in setting standards to think in terms of 
mean square error (MSE). 

The MSE of an estimate x’ of a parameter x 
was defined earlier as 


MSE, = ox + B 7 


where ox.” is the overall variance of x’, and By,” 
is the square of the bias; bias being the differ- 
ence between the expected value of x’ and the 
defined target parameter x. Mean square error 
can be expressed in several alternative forms. 
One that is instructive, and convenient for the 
discussion that is to follow ts: 


Noe = os + om + B 


where: os iS “Sampling” variance or the 
variability among possible sam- 
ple estimates that arises be- 
cause not every unit in the pop- 
ulation is included in the sam- 
ple, 
om: IS ‘‘mMeasurement’’ variance, or 
the variability in the enumera- 
tion or measuring process that 
would arise in repeated trials of 
the survey, even for identical 
samples of units, and 
B is bias as defined above. 
Where these terms are divided by the square 
of the expected value of the estimate, x’, (or 
more loosely by x"), the absolutes are convert- 
ed to relatives. The usual terminology is 


— = V2 = sampling relative variance, or 
Sampling relvariance 


—_ = V,,? = measurement relvariance, and 


—; = square of relative bias. 
The square roots of relvariances are called 
relative standard errors. 

Under ideal circumstances good survey de- 
sign will have obtained the best answer which 
circumstances permit, to a very difficult ques- 
tion: namely, the allocation of available re- 
sources to (1) sample structure and size, (2) 


methods for reducing measurement variability, 
and (3) methods of limiting bias, all in such 
a manner that mean square error is minimized. 
The solution will have included speculatively 
determined targets for permissable average 
bias in the interviewing process, and in vari- 
ability of performance among acceptable inter- 
viewers. Pre-testing may have strengthened 
these speculative determinations. The task in 
selecting and training interviewers is to meet 
these specifications as closely as possible for 
a few of the most important items in the survey. 
Under these ideal circumstances, that means 
training until the average performance of those 
interviewers who are to be retained differs from 
perfection by not more than the specified bias; 
and it means dismissing from employment or 
retraining those interviewers whose performance 
differs from the average by more than about 
three standard deviations when all candidates 
for the job are arrayed in a frequency distribu- 
tion according to their respective performance 
scores. This ideal state of affairs is rarely if 
ever fully satisfied. In practice, an approach 
which is guided by the model just described 
can do much to assure that a reasonably ade- 
quate set of interviewers is chosen to initiate 
data collection. (See Appendix B for a sug- 
gestive illustration of how one may proceed.) 
In Section 7 below, procedures are recom- 
mended for maintaining control over subse- 
quent performance of interviewers. 

7.5 Dress Rehearsal. The survey has now 
reached the stage at which it is nearly ready 
“to go." Planning of operations has been com- 
pleted, instructions written, personnel hired 
and trained, other resources and materials pro- 
vided, pretesting of questionnaire and inter- 
viewing accomplished. Plans for editing, cod- 
ing, data processing, and tabulation also have 
been set, although later sections of this manual 
will treat control of data reduction in more de- 
tail. A final dress rehearsal operation is advis- 
able before entering into formal full production 
of the official set ef survey data. 

The dress rehearsal should be a close ap- 
proach to a trial or experimental conduct of the 
final survey itself, except carried out in minia- 
ture and over a limited period of time. The ob- 
jectives include again those discussed above 
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in section 7.2 dealing with pre-testing, although 
in this step it is hoped that already proven pro- 
cedures are being tested, so that few further 
corrections are necessary. In the dress re- 
hearsal there are the additional objectives of 
testing and revising as indicated the planned 
administrative and logistical features of the 
project, and of preliminary validating of data 
processing procedures. 

The appropriate scale, extent, and detailed 
content of the dress rehearsal will vary with 
these same factors in the main survey. The 
greater the main survey, the more comprehen- 
sive the dress rehearsal should be. Remember 
that the central concept is to approximate the 
operational conduct of the main survey on a 
smaller scale to discover if the plan is feasible, 
to remove ‘‘bugs’” which are found in actual 
trial, and thus to embark finally with machinery 
which has been “‘road tested.’’ This is the same 
concept which has proved successful in manu- 
facturing industries around the world. 

When the main survey is a major undertaking, 
the ideal course is to run the dress rehearsal 
through one full trial time period or cycle of the 
pattern of the survey itself. For example, if the 
main survey is to be a continuing operation, or 
if it is to extend over a year or more of data 
collection, it likely will have been designed 
with subsamples of places or of months (or 
quarters) of data collection, or of both places 
and time. The dress rehearsal might then cover 
all—or perhaps a random fraction — of one 
corresponding subsample, or one month of trial 
collection. Data from the dress rehearsal will 
be collected, transcribed, edited and coded, 
and possibly tabulated as planned for the main 
survey, but they will not be incorporated into 
final survey results, except perhaps as prelimi- 
nary and provisional data. This is the last Op- 
portunity to correct procedural instructions for 
collecting and processing the data: to revise 
editing, coding and classification structures: to 
introduce devices to compensate for earlier 
omissions in instructions. 

Managers will be observing the dress re- 
hearsal on the administrative and logistical side 
for feasibility and for adequacy. Particular at- 
tention will be given to time tables, costs, trans- 
portation and communication problems, trans- 
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mittal and transcription of data, flow of supplies, 
personnel management, and channels of super- 
vision. Each of these elements can be critical, 
but perhaps the most important of all are the 
latter two. Lines of authority and supervision 
must be both clear-cut and firm. A disciplined 
but happy crew of interviewers and other agents 
is an absolute essential to success. Every inter- 
viewer should feel this discipline, but he should 
also believe that the system is not oblivious to 
his welfare, and does understand his prob- 
lems. He should know where and how to turn 
for help when help is needed. 

7.6 Systems of Controls over Field Collec- 
tion. The main survey is underway. Planners 
have done all they can to prepare for a sound 
undertaking. How can it be kept on the track? 
Desire, motivation, conviction, dedication, per- 
severance, courage, and careful attention to 
task on the part of all personnel are needed. 
The extent to which they are present will largely 
determine the answer. Fortunately, there are a 
number of tools which can facilitate the job. 
Again we note that the choice of tool will vary 
with type of survey, and particularly with the 
mechanism for primary collection of data, e.g., 
whether it be interview of persons, transcrip- 
tion of records, observation, or new physical 
measurement. In this section, we concentrate 
on the interview instrument, and rely heavily on 
the power of replication as a control device. 
The essence of recommended control proce- 
dures can be adapted to other data collection 
procedures — for instance, transcription of 
records — without great difficulty, and indeed 
often without any substantial change. 

For operations of the general type of the 
West Dakota survey previously discussed, rec- 
ommended controls over data collection fall 
into six classes. 

A. Control over sample inclusion, exercised 
through re-selection of subsamples. 

B. Control of the interviewing process and 
of individual interviewers through reinterview of 
a subsample of respondents. 

C. Control of the interviewing process and 
of individual interviewers through observation 
by supervisors of a sample of interviewer as- 
signments. 
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D. Control of the interviewing process and 
of individual interviewers through edit checks 


for performance. 


E. Minimization of nonresponse through a 
system of surveillance of nonresponse rates, 
assigned quotas, and, if necessary, introduc- 
tion of additional follow-up procedures. 

F. Screening early survey results for inter- 

nal consistency and correspondence with any 
available external data, and intensifying efforts 
in any area of apparent discrepancy. 
Classes A and E should be a part of all sur- 
veys. The other classes of control may be used 
individually to greater or lesser extent, or in 
combination. But whatever specific overall 
methods are adopted, they should include con- 
trol over the four factors of sampling, non- 
response, the interviewing process, and perfor- 
mance of individual interviewers. 

7.7. Sampling Controls. Validation of sam- 
pling operations has two phases. The first is 
verification of the mechanics of selection of 
first-stage primary sampling units (PSU), and of 
any intermediate sampling units for which 
sampling is carried out in central offices. The 
second relates to ultimate stages of sampling 
performed on the field by enumerators or inter- 
viewers. We treat central office sampling con- 
trols first. 

This control is essentially a verification of the 
mechanics of selection. It can be accomplished 
through either of two approaches or variants of 
those approaches. In one scheme, the work of 
sample selection of PSU’s (and intermediate 
units, if they are a part of the design) is redone 
independently by a second clerk or group of 
clerks. Any discrepancies between the two 
selections are reconciled by a supervisor, or if 
necessary, by the managing statistician. Possi- 
ble difficulties may involve only clerical errors 


or such things as cumulative impact of round- 


ing errors, or misinterpretation of an ambiguous 
instruction. But they may have more hard-to- 
resolve origins. For example, if the PSU's are 
defined in terms of geographic and political 
boundaries, it is not unlikely that there will be 
instances in which proper boundaries are un- 
certain. These situations must be resolved in 
a manner that does not bias the sampling. If 
any part of the sampling is from listed frames, 


Care must be exercised that the target universe 
is exhausted by the lists used, or not partially 
duplicated. It is easier than one might at first 
Suppose for clerks to miss one of 24 file draw- 
ers in drawing sample cases, and especially if 
one or more of the drawers is a ‘‘supplement”’ 
to the main file, a group of cases ‘“‘inactive”’ or 
“in process,’’ or in some other respect different 
from the central frame. When there are unde- 
tected duplications in the frame, the realized 
probabilities are different from those intended, 
and subsequent messy field episodes and com- 
putational adjustments must be faced. 

A second approach may be used alternative- 
ly, Or in Conjunction with the dependent verifi- 
cation just described. It is easy to apply in 
some situations, and much more difficult, or 
perhaps impossible, in others. In this approach, 
one makes calculations to see if the realized 
sample does indeed yield a statistic which is 
a satisfactory estimate of a criterion parameter 
of the target universe. The procedure is out- 
lined with the following illustration. Specific 
details of application will differ with particular 
designs, and depend upon the nature of avail- 
able data. The chosen illustration is a simple 
One, in order to more clearly display the high- 
lights of the method. 

A. Assume that the universe consists of N 
PSU’s with z; being the criterion measure for 
the i PSU. The z; statistic may be — and pref- 
erably is — a known value from some previous 
statistical or administrative program. For in- 
stance, it might be the head count in an earlier 
population census; or the numbers in a regis- 
try; or tax records; or even estimated or guessed 
numbers of children born. The zi measures 
must be in specific numbers, but it is not neces- 
sary that they be accurate measures. If a choice 
is possible, it is better to choose a Zi;-measure 
which can be expected to be positively corre- 
lated with x;-values, where x; is an (unknown) 
key statistic and objective of the survey. If no 
z, real data can be found, it is still acceptable 
to assign arbitrary 2; values to the PSU's, choos- 
ing zi numbers which one suspects are corre- 
lated with the unknown x;'s. This assignment 
of synthetic data to the PSU's can be done with 
considerable sophistication, using the same 
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type of evidence that was given weight in de- 
signing the sampling plan, taking into account 
expected mean values and speculated popula- 
tion variances, with specific selected 2; Ss being 
chosen randomly from appropriate distributions 
of z;-values. A less elaborate procedure which 
bears a resemblance to the outlined one may 
well be adequate. The assignments must, of 
course, be made by someone who does not 
have knowledge of the specific PSU's which 
have been selected in the sample. 

B. Forthe n PSU's in the sample, transcribe 
the respective z;-values that have been chosen 
in steo A. The assumption implicit in this new 
step is that subsequent stages of sampling 
within the selected PSU’s either would yield 
precise measures of z; for the PSU, or, since we 
are attempting at this point to verify only the 
first stages of sampling, the subsequent sam- 
pling is not of immediate concern. 

C. Using the z;-data for the n sample PSU's, 
and the full intended estimating procedure of 
the survey design (minus the within-PSU ele- 
ments), calculate z’, an estimate of z, the aggre- 
gate total value of the criterion statistic for the 
full target universe. 

D. The discrepancy, d = z’ — z, should 
come from the normal distribution with mean 
zero, and variance oz”, which can be calculated 
from the assigned values z; in the full universe, 
or estimated from the assigned z;-values in the 
sample. 

E. lf d is less than oz, sampling in the 
first stage can be considered verified. If 
Zou Sd S 2oz,, the sampling may be correct, 
but merits some further spot checking. if 
d)>3e,, probably errors have been made in 
sampling, and a very careful review is war- 
ranted. 

The second phase of sampling control is 
concerned with selection of ultimate sampling 
units in the field by enumerators or interviewers. 
Control of this process is best handled in com- 
bination with the Class B control system, i.e., 
through re-sampling and reinterview of a sub- 
Sample of assignments. For convenience in 
discussion, the entire process is designated the 
control subsample and also will be referred to 
at times simply as the reinterview. 
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7.8 The Control Subsample. The primary 
purposes of the Control Subsample are the 
three already identified: 


A. Confirmation of field sampling and 


proper inclusion of ultimate units in the sample, 

B. Surveillance of the interviewing process 
(both as a mechanism for formal process con- 
trol and as a motivating force for interviewers 
to perform in a manner which will pass inspec- 
tion), and 

C. Control over individual interviewers, 
identifying those who are performing satisfac- 
torily, those who need retraining on selected 
matters, and those who must be replaced. 

In addition, the reinterview may serve two 
other purposes: | 

D. Strengthening the capability of field 
supervisors, and 

E. Providing evidence which later will assist 
analysts in interpreting survey results, and in 
estimating overall mean square errors. 

It should be said immediately that operation 
of a control subsample is itself a difficult mat- 
ter. It requires honest and diligent effort by 
field supervisors. It calls for careful and tight 
scheduling. It will lose much of its value if the 
reinterview and processing of evidence from 
the reinterview do not follow closely in time 
the original interviews. There must be prompt 
feedbacks from reinterview findings to produc- 
tion operations if the control is to have full 
effect. 

The general plan of the control sample is 
simple and straightforward. It consists of (1) 
drawing a subsample of assignments to inter- 
viewers; (2) re-doing by experts the work that 
had been intended for the subsample of assign- 
ments; (3) comparing the results from the 
Original assignments with those from the check 
assignments for sample coverage, data con- 
tent of key items, and for variation among inter- 
viewers in performance; and (4) taking any 
indicated action to bring the main survey into 
closer conformance with the survey design. 
While the strategy is simple, the tactics or de- 
tails of operation of the control subsample are 
not amenable to simple or unique specifica- 
tions. An illustrative arrangement is offered be- 
low, with attention being called to several par- 
ticularly significant aspects. 
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Suppose field work is built around the follow- 
ing elements. (The specific numbers mentioned 

for each parameter are included in order to 
~ make parts of the illustration more concrete. 
They are not necessarily recommended quan- 
tities): The discussion makes use of an impor- 
tant design and operational element called an 
“assignment.’’ An assignment is a_ specific 
piece of work given to an interviewer (or reinter- 
viewer) to be accomplished in a stated period 
of time — often a day or a week in survey pro- 
grams. Not uncommonly, it will consist of 
collecting data from a cluster of 10 to 25 sam- 
pling units in geographic proximity to one an- 
other. 

M = number of interviewer assignments for 
one time period or cycle of the main 
survey — perhaps a 3-month period — 
perhaps 500 assignments. 


H = number of households in the main sam- 
ple in this period — perhaps 10,000. 

N = number of persons in the main sample in 
this period — say 40,000. 

H = H/M.= average number of households 
per assignment in main sample — 20. 

N = N/M = average number of sample per- 


sons per assignment — 80. 
K = number of interviewers — say 50. 
M = M/K = average number of assignments 
per interviewer in the time period — this 
can vary widely from one survey to an- 
other: for the illustration it is 10. 
number of expert reinterviewers—say 10. 
number of interviewer assignments in the 
control subsample. 
h = number of households in the control or 
reinterview subsample. 
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n = number of persons reinterviewed. 

h = h/m = number of households per as- 
signment in the reinterview. 

n = n/m = average number of reinterview- 
ed persons per assignment. 

n = n/h = number of persons per house- 
hold reinterviewed. 

m = m/K = average number of reinterview- 


ed assignments per interviewer. 
It is assumed that work loads are such that all 
expert reinterviewers — probably the field 
supervisors — participate equally in the reinter- 


view procedures, each handling K/R = 5 
interviewers. 

A number of decisions must be made; most 
of them being subjective judgments since firm 
planning data which would permit more rigor- 
ous determinations will be lacking. Clearly one 
decision must be to make m, A, and n large 
enough to provide sufficient data to be useful 
for any purpose. The sizes of m, h, and n are 
limited by several factors, and especially by 
cost and by the demands on the time of super- 
visors. It is certainly to be hoped that most 
Original interviewers will miss not more than a 
small proportion of persons who should be in 
the sample. This means that for the coverage 
check, a moderately large subsample is needed 
to detect a significant omission rate. 

A start toward determination of subsample 
‘‘size’’ might be made in this way: Perhaps a 
tolerable omission rate is 5%. Managers wish 
to make reasonably sure that if the actual 
omission rate is double that, it is not measured 
as low as 2%, and thus incorrectly accepted. 
Translate ‘reasonably sure’ to mean there 
should be only a 5% chance that an effective 
probability of omission 0.04 is measured to be 
as low as 0.02. The control procedure is con- 
cerned with overcoverage as well as under- 
coverage, but since undercoverage is the more 
serious problem, in planning one is reasonably 
sure that if he provides adequately for detect- 
ing omissions, he will do at least passably well 
on the other side. While some persons may be 
missed within a household, the important risk 
of omission is of entire households, and so 
subsample size is approached initially in terms 
of households. Assuming as an adequate model 
simple random sampling, relevant calcula- 
tions are (with P = probability of omission, 
Q=1 —P, o? = variance of estimated P, 
h = number of households): 


1.650 = 0.04 — 0.02 


o ~ 0.012 
_ PQ _ 0.04(0.96) _ 4, 
hmmm O00T44 ame 


This suggests that it will require a subsample 
of approximately 260 households or 13 full 
assignments to meet the suggested coverage 
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check. It is immediately clear that the control 
subsample cannot provide in one cycle an ade- 
quate coverage check for each interviewer, for 
that would require more than a 100% reinter- 
view. But if reinterviewing is spread out evenly 
among supervisors, interviewers and time, one 
reinterviewed assignment per interviewer would 
easily meet the overall coverage check each 
month. This suggests a tentative plan for re- 
check of five assignments by each supervisor 
each cycle, covering one assignment of each 
of the interviewers under his supervision every 
three months. This is a reasonable work-load 
for the supervisor, and it has the attractive ad- 
ministrative simplicity of one check per inter- 
viewer each cycle. The latter point exhibits one 
of the intrinsic strengths of the reinterview pro- 
gram. Assuming that the choice of recheck 
assiqnment for an interviewer is made random- 
ly from among each 10 of his assignments, the 
interviewer is under continuous risk of having 
his work checked, and thus is motivated at all 
times to do a better job. The incentive is espe- 
cially obvious if the interviewer's compensation 
is influenced at all by the quality of his perfor- 
mance. 

How does this tentative scheme fit other ob- 
jectives? Some further procedural questions 
must be answered first. The full assignment 
was checked for coverage because it is proper 
inclusion of households that is the most critical 
element of the check. For reasons of minimiz- 
ing travel cost, and completing the coverage 
check of persons, it is desirable to also reinter- 
view all households in the recheck assignments 
for sample membership. The situation with 
respect to detailed reinterview of persons for 
subject content is different. Here several new 
questions arise. In the original interview one 
respondent may have replied for all members 
of the household. There is likely to be an intra- 
household correlation in accuracy of reporting 
for different household members, and some of 
the data applies to the household as a whole 
rather than to individual persons in any event. 
It appears advisable to conduct the reinterview 
with respect to only the household questions 
and one subject person per household. 

Thus the tentative plan embraces interview 
of only 1000 persons per cycle, an average of 
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20 respondents per interviewer, and 100 re- 
spondents per supervisor. Completed inter- 
views will be a bit less than these figures be- 
cause of unavoidable non-response. 

A number of variations and modifications of 
the above plan will occur to the statistician 
faced with a particular set of conditions. In 
evaluating a chosen scheme, it is especially 
important to realize that the scale of activity in 
the contro] subsample must for reasons of cost 
and feasibility be modest. There will at few 
points be a sufficient number of observations to 
be absolutely sure of conclusions. Accordingly, 
the reinterview procedure must be considered 
as only a part of the overall control of field 
operations. It can do a great deal, but not the 
whole job. 

This manual is not a textbook on formal pro- 
cess or quality control procedures as they have © 
been developed in industrial production. The 
student who is interested in optimal designs 
and in more elaborate interpretation of the evi- 
dence coming from those designs is referred 
to one of the several good textbooks — for 
example, references [4] or [7]. Some addi- 
tional comments on formal quality control are 
offered in section 8 below, ‘Data Processing.” 
A few notes on formal quality control are pre- 
sented in Appendix A. A thorough grasp of 
formal quality control theory would permit a 
Survey manager to introduce some refinements 
and efficiencies into control of the interviewing 
Process and of interviewers, but is by no means 
necessary. A more pedestrian approach is 
taken in here. This course is justified by three 
main considerations. First, the body of theory 
which underlies formal quality control designs 
is extensive, highly mathematical, and complex. 
It is a field in itself. Thorough understanding 
requires either very considerable Study or al- 
ready trained personnel who may not be avail- 
able. Second, the nature of the interviewing 
process is such that exact precision of measure- 
ment and determination of parameters needed 
for most formal quality control models is not 
obtainable. A greater element of subjectivity 
enters the picture. Third, and perhaps most 
important, the design of an effective reinter- 
view program is determined better by careful 
attention to administrative factors and opera- 


tional feasibility, than by too slavish adherence 
to a plan that promises but still may not be able 
to deliver some specified numerical level of 
accuracy. 

Indeed, the indirect effects of quality control 
procedures such as reinterviewing and repli- 
cate coding are likely to be more important 
than the direct effects. For as noted earlier, 
the knowledge by interviewers and coders that 
their work is to be checked is a powerful in- 
centive to do the job correctly in the first place 
— and certainly to avoid deliberate faking or 
‘“curbstoning,’’ a term sometimes used to refer 
to the dishonest action of an interviewer who 
sits at home or on a curbstone and fills out a 
questionnaire without actually talking to the 
named respondent. 

We proceed to comment further on how evi- 
dence from the control subsample described in 
this section can be utilized. 

For operational purposes, a simple model 
provides an adequate key instrument for evalu- 
ating performance, and suggesting appropri- 
ate action — whether the focus of attention is 

-on a Single interviewer, the group of interview- 

ers under a common supervisor, or On a par- 
ticular question or set of questions in the sur- 
vey. Let n. be the number of observations of 
the item or category c under study; d. be the 
number of errors observed for that item (i.e., 
the number of instances in which the reinter- 
viewer's finding differs from that of the inter- 
viewer). Then P.’ = d-/n. may be treated as an 
estimate of the probability of error for that item 
for the category of n. observations being con- 
sidered. The estimate P’. can be assumed to 
have a simple binomial distribution with vari- 
ance o'r, = Pit) -P.')/ne. Consider, now, 
some of the uses of this tool. 

It will be convenient to assemble data from 
the control subsample in tables which have a 
standard format, similar to that illustrated in 
Table 7-1. Table 7-1 is a generalized table, 
illustrated for evaluation of the reporting of 
numbers of persons with one or more chronic 
conditions. It can be easily adapted to the 
coverage check, or to proper reporting on any 
specified topic, group of questions, or classes 
or types of respondents or subject persons. 


Tables like 7-1 can be consolidated, and 
summed over interviewers, reinterviewers, ques- 
tion categories, or time intervals. 

The symbols appearing in the column heads 
correspond to those in the simple model just 
described. Note that the table allows for two 
counts by the reint:rviewer, yc.’ and ye, the first 
his initial determination, and the second his 
final determination after ‘‘reconciliation.’’ While 
there are acceptable variations, the table pre- 
sumes a procedure in which the reinterviewer 
first records his finding without knowing what 
the original interviewer reported; then consults 
the report of the interviewer; compares the two, 
and reaches a final conclusion, ye, perhaps 
consulting further with the respondent, and per- 
haps amending his own initial finding. This 
procedure is recommended for these reasons: 
(1) Although fully independent verification 
procedures are preferred for mast operations, 
in the reinterview situation, the presence of the 
reinterviewer at the location of the respondent 
at the time of reinterview offers an opportunity 
for reducing misunderstandings through the 
reconciliation process. (2) The reinterviewer, 
although a presumed ‘‘expert’’ may be little if 
any more skillful than the better interviewer, 
and may properly be influenced in making his 
final judgment by taking account of the opinion 
of the interviewer. (This does put some strain 
on the integrity of the reinterviewer who is the 
interviewer's supervisor, but higher level super- 
vision is likely to detect misbehavior in this 
respect should observed differences over an 
extended time range be too much smaller than 
expected.) (3) The reconciliation steps are a 
valuable training device for the re-interviewer, 
in making him more intimately aware of the 
difficulties which interviewers are having. 

The table includes both the algebraic differ- 
ence, d-, and the absolute value |d.|. This per- 
mits summation to reflect either gross or net 
differences, and, of course, points the direction 
in which re-training or other corrective actions 
should move. . 

As noted earlier, when the focus of attention 
is on an individual interviewer, a single cycle 
of reinterview (in the illustration), offers only 
crude evidence of performance. When the sta- 
tistic of interest is a person characteristic, the 
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single cycle provides only 20 observations. An 
estimated error rate of 30 per cent, for example, 
has a sampling error of 10 percentage points. 
A 30 per cent error rate is intolerable, and 10 
percentage points in sampling error leaves too 
much doubt. But consolidation of evidence 
can improve the picture markedly. The inter- 
viewer who makes anything like 30 per cent 
error on one item probably has errors of similar 
scale on, say, 10 other items. In that event, 
the sampling error for the group of items is 
cut to about 3 percentage points, and it be- 
comes pretty clear that the interviewer is per- 
forming in an inadequate manner. Combining 
observations for the interviewer over several 
cycles of reinterview has a similar utility. Com- 
parison of the estimated error rates among 


Table 7-1. 


interviewers may well isolate the poor perform- 
er. Use of this rather subjective pattern of 
evaluation in conjunction with the simple bi- . 
nomial model described above, is the approach 
recommended in this manual. When an inter- 
viewer shows unsatisfactory performance, he 
must be retrained — and that is preferable, if 
feasible — or replaced. 

Some may prefer more structured guidance 
for interpretation of reinterview data. Such is 
offered below, with the caution that it still has 
arbitrary and subjective features. It is a product 
of standards which could be set at different 
levels and is an adaptation of an ‘acceptance 
sampling plan.” 

Step One. Identify Sections |, Il, Ill,... of 
the questionnaire, for which separate surveil- 


Re-Interview Data on Number of Persons with One 
or More Chronic Health Conditions for District A, West Dakota Survey, Cycle 7 


For Subject Persons Reinterviewed 


Stub Number 


Persons Bees: 
Interviewer 
c n. x 
(1) (2) (3) 


Interviewer 1 
Interviewer 2 
Interviewer 3 
Interviewer 4 
Interviewer 5 


or 
Respondent Type 1 
Respondent Type 2 
Respondent Type 3 
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Number Persons with one or more chronic conditions 


or 
Question 1 
Question 2 
Question 3 


Reported by Reinterview Difference 
; After i. 
Initially Reconciliation Xo-Ye Ix.-ycl 


(6) 7) 


lance is desired. For example, some of the 
sections might be: 


section | Coverage of households. 

section Il Coverage of persons. 

section Ill Questions on demographic 
characteristics of individual 
persons. 

Section !V_ Health or Labor Force 
attributes of persons. 

Step Two. For each section, identify and 


count the units of observation, i.e., the effective 
sample size — usually number of households, 
number of persons, or number of person-ques- 
tions. (This is ne for a category.) 

Step Three. For each section.select an 
approximate level of acceptable error rate, i.e., 
a target value of P,.. 

Step Four. Define as Acceptance Number 
the maximum number of permissible differ- 
ences, d., for a given range of observations, Ne, 
for each section. 

Step Five. Build a table like Table 7.2, 
‘Acceptable Difference Limits” for each section. 

Step Six. A decision is made to ‘‘accept”’ 
or ‘reject’ the work of each interviewer for 
each reinterviewed assignment. If the work is 
accepted, no further action is taken. If rejected, 
the interviewer must be retrained on the re- 
jected section(s), or replaced. 

Table 7.2 was built around a target accept- 
able quality level (AQL) of 5 per cent error rate. 
This means that if the true average error rate 
is 5 percent or less, the performance will be 
accepted at least 95 percent of the time. The 
figures in the right-hand column are approxi- 
mations to what is sometimes called ‘‘the con- 
sumer’s risk.’ They are interpreted in the 


Table 7.2 Acceptable Difference Limits for 
Section IV 


(For example, number of persons with specified 
health attributes) 


Acceptable 

Observations Ditlerences ete 
10- 29 z 26% 
30- 49 4 20% 
50- 69 6 18% 
70- 99 7 14% 
100-129 fe) 13% 


following way: If, for example, there are 60 ob- 
servations, and the acceptance number 6 is 
used, an actual underlying error rate of 18 per 
cent or greater will not be accepted on more 
than 10 per cent of inspections. It may appear 
that the acceptance numbers in the table seem 
to reflect error tolerances which are nearly 
double the target specifications. The explana- 
tion lies in the fact that the acceptance numbers 
are maximum values. Observations with fewer 
errors will also be accepted, and so the average 
error rate will be smaller. 

Those who might wish to set different stan- 
dards in constructing a table similar to Table 
7.2, may consult quality control textbooks. (cf. 
p. 358ff of reference [7] or p. 154 of reference 
[4]. 

An analogous procedure to that outlined for 
evaluating interviewers can be employed to 
assess whether a particular question or respon- 
dent type, or a group of questions or respon- 
dent types is being handled in satisfactory 
fashion. 

It will be desirable for the purpose of sur- 
veillance of the overall interviewing process to 
construct and maintain a simple control chart. 
The chart gives a picture of the trend of a sum- 
mary error rate over time and tells the survey 
manager whether the process is ‘“‘in control.”’ 

Again there are alternative routes for estab- 
lishing and using the control chart. The follow- 
ing scheme is straightforward, and provides an 
informative tool. 

Step 71. Form a weighted average of the 
chosen target error rates over all sections of 
the questionnaire and all interviewers. Average 
the error rates among interviewers within ques- 
tionnaire sections for a single cycle of the 
Survey, and then form the weighted average of 
these section averages to yield 


i 
R= > WiA; 
ti 


Where fF; is the average target for all inter- 
viewers for section / of the ques- 
tionnaire and interview process; 

w; is the assessed relative importance 
of section / in the overall process. 
For example, wi may be assigned 
the value 0.30 where / is the cov- 
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erage process, Wi assigned 0.10 
where / is the demographic attri- 
bute section; wi assigned 0. 20 for 
the health attribute section,... 
Swi = 7.00; 
L is the number of sections; and 
R is the overall target error rate. 
Step 2. For each cycle of the survey, com- 
pute the realized overall error rate R’ with the 
formula 


where R; is the observed average error rate for 
the section, for a given cycle. 

Step 3. Assuming that the interviewers have 
approximately equal work loads, an adequate 
estimate of the sampling variance of an ob- 
served error rate Fi’ is 


Ry’ 7 ‘ae Ry’ 
Sea. te 


where ni is the total number of observations 
over all interviewers for section / for one cycle. 
For example in the illustration on p. 25 for 
household coverage, n will be approximately 
1000. 

The variance of the estimate of realized over- 


all rate, R’ will be approximately S*x, = XwiS*p.. 
The standard error of R’ is the square root Sr. 
Step 4. Set up the control chart as in Figure 
7.1, and post the values of R’ to the chart each 
cycle of the survey. 

_ ‘The ¢enter horizontal heavy line is the over- 
all target acceptable error rate. The horizontal 
lines at R eISe, are the boundaries of tolerable 

’ observed error rates. As long as the plotted 

values FR’, fall between these boundaries, the 
the interviewing. process can be considered “‘in 

~ control.” In the illustration in Figure 7.1, the 

_ Steady climb in R* from cycle 6 through cycle 
9 raises a suspicion of emerging trouble, but 
the process has Still remained in control through 
cycle 9. -At ‘cycle 10, however, it moves out of 
control. At that point, prompt and firm action 
must be taken. The components of the weight- 

ed average F’ which are principally responsible 

for the high error rate must be located, and 
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corrective measures taken to bring the process 
back into control. 

Evidence from the control subsample can 
serve one further important function. As the 
analyst studies final tabulations from the over- 
all survey, attempts to draw conclusions and 
write reports for presentation to officials or for 
publication, his work will have greater validity 
if he is able to make sound judgments concern- 
ing the general reliability of specific statistics. 
Knowledge of sampling errors is a great help, 
and it is presumed he has that information. The 
reinterview data can give him another powerful 
tool, for they provide an indicator of the sta- 
bility of the measurement process. This tool 
is not a precision instrument, but it is sufficient- 
ly sharp to permit the analyst to distinguish the 
consistently measured statistic from the one 
with highly unstable response — and conse- 
quently one with questionable validity. Data 
from a table similar to Table 7-1 provide the 
needed input. For a particular statistic — say 
percentage of persons who have had a speci- 
fied disease — one can calculate for each in- 
terviewer for each cycle the rate r = x/n as 
estimated from interviewer reports, and the 
rate r’ = y/n as estimated from reinterview re- 
ports. The difference D = r — r’ is a first 
crude measure of consistency or stability in 
measurement of this item. The average squared 
value of this difference taken over all interview- 
ers and all cycles, 


where the subscript k refers to observation 
for one interviewer in one cycle, and 
K is the total number of interviewer- 

cycles, 
is a summary measure of the stability of the 
Statistic. Statisticians will differ on the exact 
interpretation of the statistic S°, but all will agree 
that it tends to rank order items with respect to 
the consistency of measurement. If r is the 
overall survey estimate of a particular rate, 
then v = S/r is a relative measure of the in- 
consistency of measurement. One group of 
Statisticians at the U. S. Bureau of the Census 
has developed a more eiaborate model for 


analyzing reinterview data for consistency or 
stability of particular statistics. See reference 
[6]. Their model begins with data being classi- 
fied as in Table 7-3. In this table, data for each 
individual sample observation are recorded: 
the value 1 is assigned to an individual classi- 
fied as having the specified characteristic, and 
the value O otherwise. 

Thus of the n individuals reinterviewed, a had 
the specified characteristic in both interview and 
reinterview, d in neither, Db in reinterview but not 
in interview, and c in interview but not in re- 
interview. The quantity (b + c) /s called gross 
merence, and (a py ~— (a +c) = (6 —C) 
is called net difference. Gross and net rates 
are (b + c)/n and (b — c)/n, respectively. 
Since either (O — 7)? or (7 — 0)? is equal to 
unity, the mean sum of squared individual differ- 


Table 7-3 Numbers of Individuals with and 
without Specified Characteristics 
on Interview and Reinterview for 
identical Individuals 


Results of 


Results of Initial Survey 
Reinterview aor ee 
Survey 1 0 Totals 
1 a b a+b 
0 c d c+a 
Totals a+b+ct+d=n 


at+c ip+Te 


ences is simply the gross difference rate 
g = (b + c)/n. (This is a measure similar in 
concept to the S? of the previous section. It 
can be shown that the expected value of g is 
equal to twice the simple response variance. 
See reference [6].) 


Figure 7.1. Control Chart, Interviewing Process 
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One estimate of the overall population vari- 
ance of the proportion of individuals with the 


specified characteristic is 
a + ) 


(ee 
ar) 


another |s 


"Ca 


and a third in their average 
S.2? = % (S;" + $2’). 
The model defines the ratio of 7/2 the gross 


difference rate g, to the estimate S;° as the 
Index of Inconsistency, 


/ = g/283", 


and interprets this index, /, roughly as the pro- 
portion of total observed variability in the pop- 
ulation which is due to instability of measure- 
ments. 

7.9. An Observation Program. Some sur- 
vey managers recommend a more highly sub- 
jective form of field control than the control 
subsample. They advise an interviewer ob- 
servation program. This manual on operational 
control prefers to view an observation program 
as a supplement to rather than a substitute for 
the control subsample. 

An observation program is a systematic 
scheduling of occasions on which a supervisor 
Or an experienced senior agent accompanies 
an interviewer on an assignment, in order to 
secure through direct inspection more insightful 
knowledge of whether the interviewing process 
is being carried out as planned, and in order 
to be better prepared to take any corrective 
actions which may be indicated. The program 
has both training and control functions. It fo- 
cuses attention on much the same matters as a 
reinterview system: completeness of coverage, 
Proper asking of questions and recording of 
answers, performance of individual interview- 
ers, apparent difficulties with certain questions 
or types of respondents. In addition, it gives 
the supervisor another opportunity to assess 
the environment in which the interviewing pro- 
cess takes place, and occasions for more inti- 
mate contact with interviewers, including dis- 
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covery and discussion of points which other 
more structured techniques might overlook. 

The scheduling of observations may take 
account of three situations. The first is an ob- 
servation of a new interviewer shortly following 
completion of her initial training. The second 
is a periodic regular observation of one assign- 
ment or a part of an assignment at perhaps six- 
month intervals. (More frequently, if there is 
no reinterview system.) The third is special 
occasions when other evidence suggests an 
emerging problem, or the need for retraining. 

The primary strength of an observation pro- 
gram should lie in its training aspect, and in 
its positive force to help the interviewer to do 
a better job. The interviewer is likely to be 
under some stress when being observed by a 
supervisor. The supervisor should be consid- 
erate — he should fill the role of teacher rather 
than policeman. Yet the process does have 
control features, for it motivates the interviewer 
to do his best, it guides the supervisor and 
manager toward improved procedures, and it 
may be a timely locator of substandard or 
worse performance by particular interviewers. 

Specific scheduling of observations, with re- 
spect to point in the survey cycle, day or days 
of the week, time of day, or frequency, will 
depend not only on the general time table for 
Survey operations and other demands on the 
Supervisor, but also on assessment of what 
shortcomings are most likely to be found, i.e., 
whether the problem is low production rate, 
high non-response rate, higher error rates on 
particular items, or general inadequacy. 

It is suggested that a supervisor's report be 
prepared for each observation, following a 
structured format. This puts a desirable element 
of discipline into the program, makes it a little 
less subjective, and provides higher level super- 
visors with materials which can be at least par- 
tially summarized. The report may be most 
easily executed on a standard form, but whether 
it is or not, it should contain items similar to 
those in the following list. Except in unusual 
circumstances, and except for a possible con- 
fidential portion, copies of the report should go 
to the interviewer, and to the central office. 

(1) Interviewer name and identification 

number. 


(2) Observor name and identification 
number. 
(3) Date(s) of observation. 
(4) Number of assignment observed. 
(5) Type of observation 
A. Initial 
B. Regular 
C. Special 
(6) Coverage of households probably 
accurate: Yes ; SiGe 
(7) Coverage of persons probably accurate: 


Yes * No 
(8) Non-Response: households 
from____ in sample. 


(9) General Performance of Interviewer: 


Unsatis- 


Accep- 
Facet Good factory 


table 


A. Personal Appearance 
and Behavior 


B. Securing Respondent 
Cooperation 


C. Following Established 
Procedures 


(10) Handling of Sections of Questionnaire: 


: Accep- | Unsatis- 
Section Good) ‘table | factory 


A. Household and Person 
Identification 


B. Demographic 
Descriptions 


C. Vital Events 
D. Labor Force Items 
E. Health Items 


The effectiveness of an observation program 
comes in considerable part from its indirect 
impact on both interviewer and supervisor. But 
it is enhanced by prompt feedback of findings 
through the survey machinery in order to bring 
about improvements where weaknesses are 
discovered. 

7.10 Error Reports from Central Editing. 
Reinterviewing and observation are strong con- 
trol mechanisms which utilize inspection and 
evaluation of samples of interviewer perfor- 
mance. One other surveillance procedure, 
which renders a more intensive check of cer- 


tain aspects of reported data. is recommended. 
Section 8 of this manual will deal with editing 
and processing controls. It will be found that 
the recommended editing procedures yield 
counts of errors discovered by editors for each 
Of a number of questionnaire items, when the 
nature of the item is such that a clerical check 
(by either a clerk or a computer routine) can 
identify an imperfection. These error counts 
should be tabulated by interviewer, and con- 
verted to summary error rates, for classes of 
questions, for each interviewer. Further sum- 
mary by classes of questions across interview- 
ers, and by supervision district, provide another 
assessment of conformity to specified rules by 
interviewers, supervisors, and for classes of 
questions. 

These error summaries, like other control 
evidence, should be distributed promptly to 
appropriate points in the survey administration 
where they become the basis for corrective 
action. When subsequent reports do not show 
trend improvement in excessive error rates, dis- 
ciplinary steps are in order, unless the error is 
SO pervasive that a revised standard must be 
adopted. 

7.11 Combining Control Evidence. Through- 
out section 7 a variety of methods have been 
described foi exercising control over the col- 
lection of data— with special emphasis on 
both the overall process and on individual in- 
terviewer performance. It is quite possible one 
of these methods might suggest that a particu- 
lar interviewer was unsatisfactory, and another 
classify him as acceptable. It would make for 
tidiness to propose some rigid formula for 
combining results from the several methods, 
which would tell the survey manager exactly 
and definitely what steps he should take. While 
that might be done, it would be unwise or even 
foolish. Rather, the manager and supervisors 
must study all the evidence which the various 
control mechanisms provide. When any one 
control device produces a substantial danger 
signal, the trouble should be investigated, for 
that is the purpose of the system. But in any 
given situation, there may be extenuating cir- 
cumstances or misleading evidence. An inter- 
viewer may have been temporarily ill; a group 
of respondents abnormally uncooperative; 
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weather too severe. There might have been lo- 
cal civil disorder; some other unusual event 
might have produced a temporary disarrange- 
ment of the public temperament; the control 
report itself could have been in error. Guided 
by the evidence of the mechanisms, and using 
all collateral information at their command, 
they will be in a position to make sound judg- 
ments which should keep data collection under 
control. Generally speaking, it is better to re- 
train an interviewer than to dismiss him — thus 
saving the cost of recruiting and completely 
training a new person who may be as poor. 
It will be better to amend or reinforce instruc- 
tions than to adopt some completely new course 
in mid-stream — and thereby introduce dis- 
continuities or incompatibilities into the survey. 
Remember: the overriding objective is to stay 
as close to the original plan as is feasible. 


8. Editing and Processing Controls 


8.1. Introduction. When editing and data 
processing begin, almost all that can be done 
in planning a good survey and in securing the 
basic data input has been done. The raw ma- 
terial has all been assembled. The task is now 
to edit, refine, process this raw material into 
clean, coded, properly classified, numerical and 
categorical measures — and to do this with a 
minimum of error. 

This is an easier task than was the collection 
of data, and the establishment and maintenance 
of controls is easier. Control procedures are 
more straightforward because the operations 
that must be performed in editing and process- 
ing are more clearly definable, it is easier to 
tell what is right and what is wrong, and the 
operations have the character of standardized 
mass production, which is more readily sub- 
jected to uniform surveillance than are the 
highly variable interviewing situations. 

Needed control devices fall into four classes, 
each of which will be discussed separately. 

A. Document flow control 

B. General editing procedures 

C. Multiple-coding of subsamples 

D. Transcription control 

8.2 Document Flow Control. This is per- 
haps the simplest control phase in the entire 
survey operation. Yet, too often it is done in a 
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sloppy manner, if at all. Unless care is taken, 
gross misadventures Can OCCUT, and they are 
inexcusable. The objective is simple. It Is to 
assure that all the data and evidence which 
have been collected and should go into tabu- 
lation and estimation are in fact processed, 
properly labeled, and accounted for, and that 
none of the data are lost. 

The first requirement is a plan for processing: 
a flow chart that tells what documents move 
from what point to which other point, and what 
operations should be performed on the docu- 
ments at each point. In order to keep track of 
questionnaires and related papers as they move 
through these channels, it is almost essential 
that each sheet of paper have an identifying 
code and serial number. Control counts of the 
various types of sheets must be established at 
points as close as possible to the time at which 
the sheets are initially posted or executed. As 
the papers move along the production line, 
these control counts must be verified frequently 
to insure that all papers which started a journey 
are still present. 

This bulk control is facilitated if documents 
are processed and transmitted in blocks or 
“batches.” The documents in a batch can be 
physically banded together in a file-folder or 
envelope along with a transmittal sheet which 
identifies the documents, their batch number, 
the number of documents in the batch, dates of 
transmission from specified points, initials of 
the clerk who is responsible for verifying the 
number of documents, and perhaps carrying 
provision for certification by other clerks that 
particular processing operations have been 
performed on the documents in the batch. A 
space for comments will be provided on the 
transmittal sheet. 

A supplementary procedure also is neces- 
sary. This is a ‘‘safety-valve’”’ provision. Clearly, 
it is not feasible to stop all production and 
search indefinitely for a document which the 
controls discover to be missing. It might take 
a long time to re-locate the missing item; it is 
possible the document has been destroyed and 
can never be found. The supplementary pro- 
cedure has three aspects or phases. 

1. Initiation of a ‘‘Supplemental Batch 
No. ’’ which contains a dummy of the 


missing item(s), describes the problem, and is 
keyed to its parent batch through identifying 
codes. The supplemental batch is then put on 
a separate track in which the missing informa- 
lion is searched for, while the parent batch 
moves through regular processing. Later, the 
Supplementary batch is processed specially 
until it ‘catches up” with its parent batch, if 
the missing material is found, or a decision is 
made that the missing material cannot be found. 
(Of course no supplemental batch is created 
until a modest effort has been put forth to make 
the parent batch intact without undue delay in 
normal processing.) 

2. The parent batch transmittal sheet is 
annotated to show that the supplemental batch 
has been initiated and to permit proper book- 
keeping at subsequent control points. 

3. At appropriate later points, the data or 
evidence from the supplemental batch is inte- 
grated with that in the parent batch. The de- 
tailed steps in integration will depend on the 
nature of the missing data, whether the missing 
material was found, and on the timing of the 
“catch up." It may be possible to re-insert the 
located missing material in the parent batch 
and proceed from that point as though it had 
never been lost. Or it may be necessary to 
adjust later at some stage of tabulation or esti- 
mation. (Should the latter occur, proper action 
is covered under /mputation in section 9.) 

8.3. General Editing Procedures. Central 
editing of data reported from the field has two 
general purposes: (1) To minimize the differ- 
ences of reporting on individual questionnaires 
to the extent that care and reasonable judg- 
ments based on the evidence at hand make 
possible, and thus to submit to tabulation data 
that are as nearly accurate as circumstances at 
this stage permit. (2) To maintain running 
tabulations of numbers of errors discovered, 
by type, and by whom made, and to feed these 
results back to the field, so that supervisors can 
be guided in reducing mistakes in later cycles 
of data collection. 

The mechanics for accomplishing these pur- 
poses will vary with the precise content of 
questionnaires, with available data-processing 
equipment, and to some degree with experience 


and the availability of collateral data which may 
be useful for checking such things as accept- 
able magnitude or range of entries for particular 
items. The principles, guidelines, and illustra- 
tions offered in subsequent paragraphs of this 
section should be directly applicable in many 
Situations, and suggestive of appropriate steps 
In others. (Formal quality control over coding 
will be treated in section 8.4.) 

A written loose-leaf manual of editing instruc- 
tions should be prepared. First drafts of this 
manual should have been developed in the 
pre-test and pilot stages of the survey. The 
manual should cover all aspects of editing. It 
is best put into loose-leaf format because addi- 
tions and changes will be found desirable as 
experience accumulates, and new situations 
are encountered. For later reference, each page 
should carry an effective date. An index to the 
manual will be a great convenience. For ease 
in handling, it may be good to assemble the 
manual into more than one book or volume. 
The manual should be quite specific and ex- 
plicit: It should direct the clerk as to exactly 
what to do in each of the commonly encoun- 
tered situations. 

A general strategy for editing needs be 
adopted. Three broad classes of faulty data 
will be found, and remedial action selected for 
6acr, 

A. Omissions — there is no entry for an 
item for which there should be an entry. 

B. Incomplete entries — part of the entry 
is present but part is not, e.g., the name of a 
village is entered, but its district is not shown. 

C. Incorrect entries — evidence either in- 

ternal to the questionnaire or external to it 
indicates that the entry cannot be correct, or is 
extremely unlikely. 
Decisions must be made for each item or class 
of items whether the clerk is expected to supply 
missing data, and if so from where to obtain 
the needed information; and whether and how 
to attempt correction of an entry judged to be 
incorrect. : 

For some items, a missing or incomplete 
entry can be supplied by the clerk with little 
chance for mistake. Examples are cycle num- 
ber; a code for an interviewer (determined by 
handwriting, or location of respondent, or ini- 
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tials on a transmittal sheet); sex of respondent 
(from respondent name, OF content of question- 
naire); number of persons in the household (by 
counting names of members); the district num- 
ber of a named village (by reference to a cata- 
log); or number of nights spent in a hospital 
(if admission date and discharge date are 
shown). For a few missing items, it may be 
feasible to secure the needed information from 
a supervisor, or even from the field — but us- 
ually this will entail operational delays which 
cannot be afforded. There are also certain 
types of corrections which the clerk can make 
with low possibility of mistake. First, are the 
cases in which a faulty entry is made for an 
item for which the clerk could have supplied 
the data had the entry been blank — cases like 
those in the examples above. There are others. 
lf date of birth and age are inconsistent with 
one another, the one with greater probability 
of error can be adjusted to match the other. 
(The better choice may vary among different 
societies. In the U.S.A., date of birth is more 
likely to be correct.) If the sex had been en- 
tered as female, but several other entries ob- 
viously relate to a male person, the sex code 
can be corrected to male. Alternatively if both 
a male and a female appear on the same sheet, 
and an obviously female health condition has 
been credited to the male, it seems reasonably 
safe to transfer the condition to the female, 
especially if the physical arrangement of the 
questionnaire makes a mistake of this type not 
unlikely. If the year of data reference is, say 
1972, but the entry reads ‘1872,”’ it can be 
corrected to ‘1972.’ If the question was, ‘‘How 
many days last week were you confined to 
bed?"’, and the entry reads ‘‘30,”’ it is probably 
good editing to change the entry to ‘‘7,”’ on the 
ground that the ‘30’ refers either to month or 
to the total episode. In any event ‘‘7’’ reflects 
a smaller error than ‘30,’ and will be less 
damaging to summary statistics. 

Another type of decision is determination of 
which editing changes should be made by the 
clerks, and which should be made by machines. 
In the first instance, a planner should think in 
terms of the logic of a contemplated editing 
step. Probably in the majority of circumstances, 
it will be concluded that a clerk should perform 


36 


the edit. But if the logic is straightforward, and 
tne number of possible corrective steps not 
large, and if suitable equipment is available, it 
may be more efficient, and more readily con- 
trollable, to perform the edit with an electronic 
computer. Examples are given in the following 
two subsections. 

Any simple arithmetic editing step is espe- 
cially appropriate for machine handling. Sup- 
pose, for example, the questionnaire had asked 
for number of days in the previous year that the 
person had been in each of three labor force 
status categories, and one report shows: 


Not in the Labor Force 180 days 
Employed 160 days 
Unemployed 90 days 


This total, as reported, is 430 days, but of 
course there are only 365 days in the year. lf 
there is no other specific evidence on the sub- 
ject, it may be appropriate in editing to multiply 
each of the reported day figures by the adjust- 
ment factor 365/430 = 0.849, and thus control 
the sum to 365. The computer can be easily 
programmed to make this edit with no numerical 
mistake. 

More elaborate computer edits are quite 
feasible. One notable class is the treatment of 
“outliers.” The frequency of occurrence of 
most events in any population follows a fairly 
regular pattern or distribution. For example the 
number of children ever born to women 20-24 
years might be reported in the unedited survey 
is shown in the tablé on p. 35. 

The reported case of 15 children is an ‘‘out- 
lier.”’ It might conceivably be a fact, but it is 
extremely unlikely and is separated from the 
rest of the smooth distribution. Even if there is 
one sample case with 15 children, inflation of 
that figure to universe proportions would almost 
surely be misleading. Theory for handling out- 
liers is Controversial. One practice, preferred 
by many, is to replace the outlier with another 
value drawn randomly from the reported distri- 
bution, excluding the outlier. An operational 
procedure for the present illustration has these 
steps: 

|. Form the cumulative percentage frequen- 
cy distribution shown in the final column of the 
table on p. 35. 


ll. Draw a random number between 1 and 
100. 

Ill. Compare this number with the cumula- 
tive distribution. Say the number is 63: since 
it is greater than 52 and not greater than 75, 
this is a ‘‘match” with the class which has 3 
ever born children. 

IV. Substitute ‘3’ for ‘'15”’ in the record of 
the outlier. The computer can perform this edit 
very rapidly when the test and correction phase 
is scheduled to follow a preliminary tabulation 
of reported data. In the illustration the pro- 
cedure would be applied whenever the reported 
number of children was greater than 9. 


No.of Women = Adjusted Cumulative 


No. of 


Children fiat tee lilacs 
of Children Number Percent 

0 10 10 t 
1 20 30 22 
2 40 70 o2 
2 30 100 72 
4 15 1D 86 
5 8 1Z3 92 
6 4 127 95 
¢ 4 ie 98 
8 2 ioe 99 
9 1 134 100 
10 @) 

a. 0 

12 0 

13 @) 

14 @) 
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As noted earlier, in clerical operations the 
clerk should have instructions which tell him 
what to do in any situation. This means there 
must be a residual instruction which tells him 
to “consult supervisor’? when no other rule 
appears in the manual. The supervisor will 
make an emergency decision and maintain a 
log of such decisions. When the log indicates 
any substantial frequency of a particular kind 
of case, an addition or amendment should be 
issued for the manual of editing instructions. 

Running counts of all editing changes should 
be maintained, whether the edit action is taken 
by clerk or by computer. Tabulations of these 


counts, as noted earlier, are sent to the field 
for action. They are also made available to 
analysts for their guidance, and in summary 
form should be included in published reports 
as a part of the tecliiical description of the 
Survey. 

The clerical edit is a control process. It is a 
review of operations done by the interviewer. 
Supervisors should give some attention to 
whether the editors themselves are performing 
in a standardized fashion. This is best accom- 
plished through two types of action. The first is 
a systematic re-edit by the supervisors of a 
Small sample of the work of each editor, and 
occasional additional spot checks of those who 
perform less well. The other comes from (prob- 
ably informal) records of relative frequency of 
finding errors by the various editors. Wide vari- 
ations among editors indicates the need for 
further spot-check re-editing, and perhaps addi- 
tional training. 

8.4 Multiple Coding of Subsamples. Usual- 
ly such attributes as sex, marital status, religion, 
race, labor force status, occupation, income, or 
diagnostic health conditions are reported in 
terms of words, checked boxes, or ranges of 
numbers. In order to be tabulated, and ana- 
lyzed it is desirable or even necessary to con- 
vert these terms into coded numbers. Here 
too the translation needs be done with mini- 
mum tolerable error. As with other survey 
operations, a plan is needed; specifications 
and instructions prepared: coding clerks se- 
lected, and trained; and a control system in- 
stalled. A detailed account of all the actions 
that should be taken, the coding guides and 
indexes written, the forms designed, the train- 
ing materials prepared, would require much 
more space than can be afforded here. But it 
should also be unnecessary for those who have 
had any experience with coding operations, and 
who have become acquainted with the spirit of 
these guidelines. 

Nor will there be presented much of the 
theory that underlies mechanisms which will be 
recommended, and no attempt will be made to 
discuss competing or more elaborate ap- 
proaches than the one suggested here. Suffice 
it to say that alternative courses do exist, and 
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they can be studied in, or adapted from, the 
literature of industrial quality control. | 

The objective of control over coding !s two- 
fold: first, to assure that the coding process Is 
under control as a process, and second to see 
that the performance of individual interviewers 
is adequate and within specified tolerances. We 
shall present in outline form an illustration of 
one well-tested and recommended structure for 
satisfying this objective. | | 

Assume that the task under observation Is 
the assignment of one of 100 possible codes to 
the health status of each sample person in- 
cluded in the survey. If it is the West Dakota 
survey, this means 40,000 codes per cycle, or 
about 160,000 codes over a year of interview- 
ing. Perhaps the reports are being processed 
in batches of approximately 100 households 
containing about 400 persons. This would rep- 
resent a block of 5 consecutive assignments 
for one interviewer in the field. It might reflect 
one day’s work for a coder. Assume there are 
9 coders who do this work, along with other 
tasks. 

The coding control system which is adopted 
is what is called a 3-way independent sample 
verification. The operating procedure begins 
with a batcn being assigned to Coder A for 
coding all 400 sample persons in the batch. 
(For convenience in discussion it will be 
assumed that all batches are of the same size, 
although in practice they will vary somewhat.) 
This same batch is later assigned to Coder B, 
with instruction to code a predesignated 10 per 
cent of the cases—the mechanics of form 
construction and posting of codes being such 
that B does not know codes A assigned. Then 
the batch is assigned to a third Coder C, who, 
independently of A and B assignments, also 
codes the same predesignated 10 per cent 
subsample. The three codes for each of the 
subsample cases are compared — either man- 
ually or by machine. In normal operation, the 
code assigned by A is the one that enters into 
tabulation, of which more later. But the com- 
Parison becomes the basis for subsequent 
control decisions and further action. Errors are 


assigned to coders in accordance with these 
rules: 
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Rule |. If A, B and C all agree, no coder is 
charged with an error. 

Rule ll. If two of A, B, C agree, but the third 
does not, the third one is assigned an error. 

Rule Ill. If all three A, B, C have different 
codes, each is assigned an error. 

Data from the operation just described per- 
mit the calculation of error rates for each coder. 
Let yi; be the number of codes assigned by 
coder / in batch /, and xij; be the number of his 
errors in that batch. His error rate for that 
batch is Rij = xXij/yij. By Summing over batches, 
an error rate for a coder over a larger trial 
period can be secured: 


XX ij 
B= 2 
Yi 
J 

By summing over both coders and batches, an 
overall process average error rate is obtained: 


The observed values Rij, Ri, and R are of course 
estimates of the true underlying error rates, 
being based on samples of nij;, ni, and n obser- 
vations respectively. Pilot studies and survey 
requirements will have established a maximum 
target overall error rate R which is both feasible 
and acceptable. Training must have been con- 
tinued until in a trial period the observed error 
rate R meets or betters the target R. The prob- 
lem then becomes one of keeping the process 
in control at that level. 

More precise and elaborate models could. be 
constructed, but a simple binomial model is 
adequate for building a control mechanism. 
This means that analysis of the observed values 
Rij, Ri, and R will be carried out under the 
assumption that they will be distributed nor- 
mally about their expected mean values with 
respective variances of 


ek ac a 
ij Nij 
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R;;, Ri, and R are all expressed as relative num- 
bers less than unity, rather than in percentage 
terms. 

As was the case in controlling the interview- 
ing process, discussed earlier two types of 
analysis will be useful One builds on the con- 
cept of ‘acceptance sampling,’ and the other 
utilizes the ‘“‘control chart technique.” 

Consider first the procedure for accepting 
or rejecting the coding on a particular batch. 
Assume that the target acceptable error rate R 
is 10 per cent. (In language previously em- 
ployed this is the AQL value.) Build a table 
like Table 8-1, which shows the maximum num- 
ber of errors, as detected by the 3-way certifi- 
cation procedure, which will permit the reten- 
tion of the batch as originally coded. If this 
number of errors is exceeded, the entire batch 
of 400 cases (not just the 10 per cent sample 
of 40 triple-coded cards) must be recoded 
(dependently) by another coder who has been 
performing at an acceptable error level. (This 
procedure makes the assumption that the 
measured error rate in the 3-way verification 
applies equally to coders A, B, and C, and in 
particular to production coder A. For the im- 
mediate purpose, this is a justifiable assump- 
tion.) 


Table 8-1 Acceptable Batch Error Rates for 
Coding of Health Conditions 
ae Acceptable ae 

Number of Number of Approximate 
Conditions Checked _ Errors in the Value of 
in Subsample Batch Subsample P.,.:0(100) 
Ni; c 
25-34 2 30% 
35-44 rg 29% 
45-54 9 28% 


Thus, according to the table, if there are 40 
conditions coded in a batch subsample, and 
7 or fewer errors are found in the 3-way inde- 
pendent verification, the batch is accepted, and 
moves on its production route. If 8 or more 
errors are found, all 400 cases must be re- 
coded, dependently. The control process 
assumes that if the true error rate is 10 per 
cent or less, the batch will be accepted 95 per 
cent of the time, while there is only a 10 per 
cent chance (Po.10) that a batch containing an 


error rate of 29 per cent or greater will be 
accepted. Reference [7] or other sets of qual- 
ity control tables can be consulted, if the survey 
managers should wish to set different levels of 
control. 

The control chart technique may be used in 
Surveillance of the performance of individual 
coders or of the overall coding operation. The 
procedure is similar to that described on page 
29 for the establishment of a control chart. It is 
illustrated now for control of an_ individual 
coder. The central horizontal line in the control 
chart, Figure 8-1, is the target maximum per- 
missible error rate R. The plotted points are 
values Ai; for the i" coder for successive 
batches / coded by him either as the produc- 
tion coder or as the ‘‘verifier.’’ The standard 
error bands are located through computation 
of average variances, using the formula 
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So long as the coder stays within the area 


bounded by the lines at R + 2Sp, he is “‘in 


control.’’ Whenever a plotted point falls outside 
this band, the work of that coder should be 
given a general review by the supervisor. If 
a second point falls above the band, the coder 
should be withdrawn from the production pro- 
cess and either retrained or replaced. lf a 
majority of points fall below the central line RA, 
and in particular when points fall below the line 
R — 25sp , the i coder is performing at a lower 


error rate than the required target. Several 
interpretations are possible. First, it may be 
expected that some coders will do better than 
others; there will be a dispersion of coder 
averages about the central R, and this is en- 
tirely appropriate. It may be that the coding 
process on the whole is proceeding at a better- 
than-expected level of accuracy. That is prob- 
ably good, and may indicate that a new value of 
R should be adopted in subsequent analysis. 
Whether this is the case can be checked by 
combining evidence from all coders, as out- 
lined in the next paragraph. It could be that 
Coder i is spending more time per case than 
was allotted for the coding operation — and 
more than is necessary — and obtaining less 
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oroduction than |s desired. The supervisor 
should be able to determine if that !s so, and 
to act accordingly. 

Surveillance of the overall coding process 
can be maintained by consolidating evidence 
from all coders, and over two or more coding 
items. The process consists of forming weight- 
ed error rates and variances, analogous to those 
described in the section on establishing an 
appropriate control chart. The plotted points 
will be the consolidated weighted error rates 
for a time period of coding — perhaps 9 to 10 
working days. 

The procedure which has been recommended 
for controlling the coding of diagnostic condi- 
tions is a 3-way independent scheme. It is 
more elaborate than a simpler 2-way dependent 
check would be. Dependent checking is ade- 
quate and effective for many operations, such 
for example, as in key punching, or in the re- 
interview subsample described in section /. 


But dependent checking Is dangerous In a Siffy- 
cult coding operation, because the second 
coder is too much influenced by the action of 
the first coder. Dependent checking lacks the 
power of true replication in evaluating precision 
in the assignment of codes. 

8.5 Imputation. Early in the present chap- 
ter, there was a discussion of several types of 
editing actions that might be taken to attenuate 
the damage caused by missing data. Even 
after editing steps of the kinds treated have 
been taken, there may well be residual in- 
stances in which particular measures or values 
are missing for some sample cases. The man- 
ner of dealing with these residual deficiencies 
is usually called “‘imputation.”’ In one sense 
imputation may be considered a part of data 
processing, but perhaps more logically it is a 
component of estimation, and is dealt with in 
the next section. 


Figure 8.1. Control Chart for Coder : on Health Condition 
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Batch Number for Coder ; (Chronological Sequence) 


9. Control of Tabulation and Estimation 


9.1 Character of the Chapter. Except for 
some remarks on Imputation, this section con- 
tains no theory, and few general principles. 
Rather, it recommends certain precautionary 
procedures, and offers some hints, designed 
to reduce the risk of computational boners. 
Mistakes will occur in transcription and in tab- 
ulation. But failure to detect and correct all 
but a trivial number of such mistakes is in- 
excusable. 

9.2 Transcription Controls. The specific 
mode of data transcription from original edited 
documents to tabulation media will vary, of 
course, with the equipment and processes 
being utilized for tabulation. Whenever pos- 
sible, the preferred mode of transcription is an 
optical, electronic, or mechanical device which 
is inherently nearly free of error. Often such 
facilities are not at hand. Then a human tran- 
scription must be made by manual copying, 
keypunching, or some similar operation. In 
such instances, some form of replicate control 
is indicated: through either 100 per cent or 
sample verification. Consider keypunching 
cards as the transcription process. If the total 
volume of keypunching is not too great, com- 
plete replicate punching may be the best 
course, using standard keypunch verifiers. If 
volumes are very large, a sample verification 
likely is better, using the same kind of plan as 
was described in section 8 for controlling the 
coding operation or a dependent verification 
similar to that described in section 7 for inter- 
viewers. 

9.3. Contro/ Totals. As a further control on 
transcription, and over a variety of subsequent 
tabulating steps, it is highly desirable that a 
series of ‘‘control totals’’ be established. These 
Start with the case and document counts which 
were established in the data processing opera- 
tions. For example, earlier steps in the West 
Dakota Survey may have established that the 
Survey consists of 40,125 households contain- 
ing 158,316 persons. Every subsequent tabula- 
tion of data which relate to households or per- 
sons should carry items of case count which 
match these controls exactly. In any instance 
in which the counts differ, a reconciliation 
should be undertaken promptly. 


The control total device can be extended be- 
yond the overall case counts. Often it will be 
feasible to establish other quantitative aggre- 
gates in preliminary simple tabulations. For 
example, the numi--r of males in the sample, 
the total number o1 virths, or the number of 
persons in the labor force may be used as sub- 
Sidiary control totals. It may even be desirable 
to tabulate a “‘nonsense”’ contro! number, such 
as the sum of serial numbers of households. 
The objective again is to have check points 
for totals which should appear in later tabula- 
tions. 

The table planning should provide for cross- 
checks from one table to another. For exam- 
ple, if one table tabulates persons by race and 
sex, and another by race and occupation, one 
step toward control should require that a check 
be made to assure that the race totals, by sub- 
Class, agree in the two tables. 

9.4 Imputation Procedures. For reasons of 
convenience in checking against control totals, 
it is desirable in tabulating to account for 
every Case, and in some manner to construct 
data for units and items which are still missing 
after the earlier editing steps have been com- 
pleted. But there is a more imperative reason 
for giving careful explicit attention to these 
imputation procedures. The first point to recog- 
nize is that when data are missing, imputation 
must take place, either implicitly or explicitly. 
If the results of the survey are to be used at 
all, the analyst or consumer is compelled to 
draw conclusions about the missing evidence. 
It is not a question of whether to impute, but 
how. 

some analysts might hesitate to ‘‘fabricate’’ 
data for missing entries, thinking that to do so 
is cheating. But suppose for a given item, 10 
per cent of the entries are missing. If no ex- 
plicit action is taken, aggregate values from 
the survey will be of the general order of 10 
per cent low, while consumers of the report 
will be asked to believe and indeed forced to 
assume that with respect to rates or ratios, the 
10 per cent missing cases have the same 
average value as the overall average of the 90 
per cent which were reported. Often this will 
be an unjustified assumption. Usually a su- 
perior imputation procedure is available. The 
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principle of the superior procedure is to ascribe 
to non-respondent or missing items the char- 
acteristics of respondents who, on the evidence 
available, are most like the non-respondents. 
The merit of this principle is evident from the 
study of a simple model for a survey of a uni- 
verse consisting of two subclasses of units, 
the subclasses being internally homogeneous, 
but with different mean values. The true mean 
value of a statistic for this universe can be 


written 
X = X/N = Pix + Pe Xe 


where x, is the mean for the i" subclass and P; 
is the proportion of universe units in the ye 
class: i.e., Pi = Ni/N, where N is total number 
of units in the universe, and N; the number in 
the i" class. When enumeration is complete, or 
when 100 per cent response is obtained in a 
probability survey, the estimate of universe 
mean Is 


x = P, e+ Pe x’. 


- 


where x; is an unbiased estimate of x; and the 
expected value of x’ is accordingly x. 

Suppose response is incomplete, with x; be- 
ing the aggregate x reported for the /'" subclass 
by ni units. If no explicit adjustment for non- 
response is made, the estimate of overall mean 
value will be 


ge a AS Se Xe 


Ny, + No n 


It can be assumed that xi/ni is a nearly un- 
biased estimate of x;, since the subclasses are 
stated to be homogeneous. Thus the estimate 
x” can be written 


The expected value of x” will not in general be 
equal to x unless the nonresponse rates in the 
subclasses are equal. But this situation can be 
improved by adopting the recommended prin- 


ciple of imputation: all data in the i subclass 
are multiplied by the adjustment factor ni/ni, 
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where n; is the designed number of units in the 
survey or census. When this is done the ad- 
justed estimator becomes: 


and once again the expected value of the esti- 
mator approximates the true value. The gen- 
eral principle can be extended, of course, to 
any number of subclasses. 

How should the subclasses within which im- 
outation is carried out be chosen? Three rules, 
taken together, provide an answer to this ques- 
tion. 

A. Choose subclasses for which the differ- 
ence between means x; and x. is, or is likely 
to be, substantial. 

B. Choose subclasses for which non-re- 
sponse rates are different. 

C. Choose subclasses which are of suffi- 
cient size (n;) that they have more than trivial 
impact on overall results, and at the same time 
assure that there will be enough responses in 
the subclass to yield a stable estimate of the 
imputed values. 

Still another reason for not using very small 
cells for imputation is that one does not wish 


any adjustment factor ni/ni to be greater than 
2.0. If a factor greater than 2.0 would be re- 
quired for some cell, it is better to consolidate 
the cell with a ‘‘neighboring”’ cell, so that the 
adjustment for the two (or more) combined 
cells is less than 2.0. Good boundaries for the 
imputation subclasses will be established in 
terms of age and sex of sample persons, and 
of geographic proximity of sample units, since 
persons of similar age, sex, and geographic 
location are likely to be more homogeneous 
with respect to many characteristics, than will 
persons in a more general population. 

For use by analysts, and by readers of the 
final report, a record should be built and main- 
tained, showing the numbers of imputations 
made for each item or class of items. 

9.5 Validation of Programming. Whether 
tabulation is accomplished by desk calculator, 


punched card electromechanical equipment, or 
computers, there must be sets of rules, instruc- 
tions, Or “‘programs”’ to direct that tabulation. 
In a major survey, these directions may be quite 
extensive and elaborate. Whether the direc- 
tions are expressed as instructions to clerks 
Or aS computer programs, they may contain 
flaws, either because they include incorrect 
Statements or because they fail to provide for 
certain contingencies that may arise. One tech- 
nique which will reduce the chance of such 
error is the use of ‘‘hot deck’’ dummy process- 
ing. In this procedure, the statistician designs 
and then creates a modestly sized set of hypo- 
thetical observations similar to those he ex- 
pects to find in the real survey. Using this set 
of data, or “hot deck,” and with very careful 
Supervision of expert technicians, he carries 
through manually a complete sequence of esti- 
mation and tabulation of a number of key sta- 
tistics. These carefully determined values 
become controls. Then the ‘‘hot deck”’ is pro- 
cessed through the contemplated regular pro- 
duction channels and results compared with 
the controls. If the results and controls agree, 
the programming is considered validated. If 
there is disagreement, the source of the trouble 
is located, and corrected. 

We speak of ‘‘designing’’ a hot deck. There 
is an art in this task. A good design will seek 
to incorporate a considerable range of types of 
cases, and at least some of the most likely 
unusual items of data, in order to provide fruit- 
ful tests of the adaptability of the programming 
to uncommon as well as to more typical ob- 
servations. The hot deck should contain such 
items as the following, to make sure program- 
ming provides for them: 

A. Blank entries where a number is ex- 
pected. 

B. Zero entries in locations in which the 
normal entry is the denominator of a fraction. 

C. A number with more digits than may 
have been allowed for the field. 

D. Numbers which, when multiplied to- 
gether, will yield an unusually large product. 

E. More complex combinations, e.g., a sub- 
traction which is followed by extraction of the 
Square root of the difference — with the hot 
deck data producing a negative difference. 


9.6 Special Gross-Tabulation Checks. 
There is little point in complete re-tabulation of 
Survey data, especially if the tabulation is ac- 
complished with electromechanical or elec- 
tronic equipment, since with only minor excep- 
tions the re-tabulation will be an exact copy of 
the original, and would provide no control. 
But there are at least two ways in which the 
general concept of replication can still be useful 
in tabulation: 

A. lf certain control totals are established 
by running an adding machine tape, these 
should be checked by a replication of the oper- 
ation before they are accepted. Other manual 
computational operations which may be a part 
of the data input, or a summary of output, or 
an integral part of data programming should 
likewise be verified through repetition of the 
operation by a second person. 

B. Perhaps the full survey encompasses an 
extensive record for each of 100,000 or more 
persons. A tabulation plan which breaks the 
overall record set into subsets, each of which 
is processed separately, has several advan- 
tages. Results from an early subset may indi- 
cate a processing deficiency which can be 
corrected before the entire survey has been 
erroneously processed. Results from different 
subsets can be compared with one another, 
with the degree of consistency serving as an 
indicator that tabulation is proceeding well, or 
perhaps that difficulties are being encountered. 
The subsets can be formed in many ways. It 
may be possible to tabulate in randomly formed 
subsets, which have the additional advantage 
of facilitating variance calculations. But re- 
gional subsets, or demographically-bounded 
domains, or parts of the long record for each 
person or unit can serve usefully. 

9.7 Comparative Analysis. Are the tabu- 
lated results reasonable? This is an imprecise 
question. It likely has no uniquely correct an- 
swer. Opinions will differ, and circumstances 
will have an impact on how one should search 
for an answer. In any event, a statistician 
should always subject his findings to this test: 
Are these results generally consistent with 
other evidence and knowledge which | possess, 
or are they contrary to expectations? 
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There usually wil be two broad classes of 
other evidence — that which is internal to the 
survey, and that which Is external. The former 
is always present in considerable degree; the 
latter will vary rather markedly from one OCcCa- 
sion to another. In both cases, the pattern of 
investigation is what may be called ‘‘compara- 
tive analysis”: evaluation following comparison 
of a tabulated figure with what might be “‘rea- 
sonably expected.”’ In the internal analysis, the 
comparisons are largely concerned with con- 
sistency relationships among subclasses and 
between related statistics. Judgments will be 
subjective; they will be based on assumptions 
such as: (1) taking sampling variation into 
account, frequency distributions and trends 
should tend to be ‘‘smooth’’ unless there is a 
positive reason for discontinuity; (2) variables 
which on an a priori basis are positively corre- 
lated, should not vary in an opposite manner 
in the survey; (3) many rates and ratios should 
tend to vary within narrow ranges among sub- 
classes — more erratic observed variation is a 
sign of trouble. 

The possible external comparisons will de- 
pend on the existence of other data which 
should show similar values. These other data 
may come from earlier surveys of the same 
type as the present one; from completely differ- 
ent kinds of measurements; from purely hypo- 
thetical or theoretical considerations; from the 
opinions of experts; or even from ‘‘common 
knowledge.’ 

It will be readily understood that compara- 
tive analysis does not require necessarily a 
close match between survey results and the 
other evidence. The other evidence may be 
outdated, weak, or wrong. Indeed, if the sur- 
vey was justified and well-conducted, the evi- 
dence from it likely is the best that can be 
found. But the comparisons should be made. 
For in some instances they may uncover flaws 
in tabulation or estimation that can be cor- 
rected, or at least recognized and treated in 
subsequent analysis. More often, they will 
focus attention on new findings, contrasts, con- 
tradictions, and highlights for which consumers 
may expect some explanation, and on which 

final reports should comment. 
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10. Control Over Report Manuscript 


10.1 Analytic Methods. Just as choice of 
formal design of the survey was not within the 
scope of this manual, neither is the choice of 
methods of analysis. The theory of statistical 
analysis is treated in many excellent textbooks 
and journals. Here we offer only one watch- 
word. Much of classical analytic theory is de- 
rived from models that assume simple random 
sampling from known distributions, with each 
observation being independent of the others 
and free from measurement error. These con- 
ditions are rarely satisfied in social surveys. 
The analyst is therefore well advised to con- 
sider critically any specific analytic technique 
before he applies it to his data, in order to 
assure himself that it is tolerably appropriate to 
the problem at hand. 

10.2 Transcription Verification. All tran- 
scription of data, codes, and labels from basic 
tabulations to worksheets, from one worksheet 
to another, and to final report copy, should be 
checked on a 100 per cent basis by someone 
other than the person who did the original post- 
ing, Or by One person reading from the source 
document, and a second person inspecting the 
posted information. 

10.3 Verification of Computations. Simi- 
larly, every computation beyond the raw tabula- 
tions should be performed twice, the two calcu- 
lations being done by different persons. Any 
discrepancies call for a third computation and 
a complete reconciliation. In addition, a vari- 
ety of additional numerical checks are desir- 
able. Computer calculations need not be so 
checked, after the programming has been 
verified. 

One valuable procedure is a series of re- 
calculations in approximate terms, performed 
by a professional statistician, to confirm the 
scale and dimension of final results. The three 
following examples are illustrative and sugges- 
tive. 

A. The initial computation was 


Moe 10.16 


35.14 ( 15.15 


) = 9789.08. 


The check, 


(8.5) 10 (1.74) 10° 


(7.5) 10 = (9.9) 10* = 9900, 


confirms the scale of the original result. 
B. The initial computation was 
12.5° + 11.97 + 14.2? + 12.92 + 13.12 = 8375. 
The check, 
5(137) = 845, 


shows that the decimal point was incorrectly 
placed in the initial computation. 


C. The original calculation showed: 


Average Number of Days 


Cycle of Restricted Activity 
__ per Person 
First 3 months ae 
2nd 3 months V2 
3rd 3 months 11.9 


4th 3 months 14.4 


The preliminary draft of the report summed 
these ‘‘quarterly’” figures and estimated 49.3 
days of restricted activity per person per year. 
The professional checker recognized immedi- 
ately that the clerks failed to realize that each 
of the cycle or ‘‘quarterly’’ estimates had al- 
ready been inflated to an annual level, and that 
the correct figure should be the average of the 
four, or 12.3 days per person per year, rather 
than 49.3. 

When a column or row of numbers has been 
converted to a percentage distribution through 
division of each number by the sum of the col- 
umn or row, the divisions can be checked by 
adding the percentages to assure that they sum 
to 100. In some cases, it may be desirable to 
adopt a special rounding rule to bring the total 
to 100. 

It is often helpful to utilize an algebraic iden- 
tity in order to set up a computation pattern 
which contains a seif-checking feature. Noting, 
for example, that 


- } 
3 (xi + 1)? = 3x2 + 25x + N, 
== 1 


one can set up a work sheet in this fashion: 


= | la xt (xi+4) | (+4) 
1 1.2 1.44 ee 484 
2 2.5 6.25 3.5 12.25 
3 
Totals _ iam fT TT IV 


Then, if the roman numerals represent the 
arithmetic sums of their respective columns, the 
check becomes: Does /V equal // plus 2 times 
! plus N? (N is the total number of observa- 
tions.) 

10.4. Completeness of Manuscript. Re- 
member that many readers of the final report 
will have no evidence on the subject other than 
that contained in the report itself — and even 
those readers who may be better equipped, will 
find it convenient to see the full story of the 
Survey assembled in the one package of the 
report. This means that the manuscript should 
be inspected to discover if it does include all 
classes of material which the reader may rea- 
sonably wish to have. This inspection should 
include at least these check points: 

A. In introductory section that presents the 
framework within which the survey was con- 
ducted, and the background for the under- 
taking. Sponsors and operators of the survey 
should be identified. 

B. Clear statement of the coverage and 
purpose of the survey. 

C. Both a condensed text statement on the 
design of the survey, and a more detailed ac- 
count (probably in a technical appendix) which 
provides a rather full description of the entire 
undertaking. 

D. A Summary of findings and conclusions. 

E. Other significant analytic results, clearly 
exhibited. 

F. Detail appendix tables, which will permit 
a reader to confirm quantitative declarations. 

G. Glossary of terms and notations. 

H. A section of the report should treat lim- 
itations and qualifications of the data; tables or 
charts of sampling errors should be included; 
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attention should be directed to possible sources 
of measurement error, and quantitative evi- 
dence on this aspect offered when known. 

|. Table of Contents. 

J. List of references, and proper keying to 
those references. | 

K. If geographic coverage or sub-universes 
are involved, the inclusion of one or more Maps 
is advisable. 

10.5 Proofreading. As many research 
workers have learned, and all people in the 
graphic arts and publications industries know, 
it is nearly impossible to convert an extensive 
manuscript to printed format without error. This 
is particularly true when the manuscript in- 
cludes equations, charts, and many figures. It 
is, neverthelsss, worth substantial effort to keep 
errors at this final step to a minimum. When 
one has devoted much thought and energy to 
operational control of the intrinsically much 
more difficult planning and production phases 
of the survey, care in proofreading the final re- 
port seems such an obvious requirement, that 
it is almost presumptuous to mention it. Men- 
tion is made here because the apparent sim- 
plicity of the proofreading question often leads 
to an expensive carelessness. 

It is suggested that in addition to an initial 
“proofing” of final reproduction copy by the 
typist, compositor, draftsman, or proofreader, 
there be at least three further proofreading 
operations: 

A. A two-person verification by professional 
personnel, one reading from manuscript, the 
other confirming the reproduction copy. This 
step should include a special operation of veri- 
fying legend and posting of any charts. 

B. An independent reading of the repro- 
duction copy by a statistician who was not an 
author of the report. 

C. A final critical reading of the reproduc- 
tion copy by the principal author of the report. 


11. Concluding Comment 


some readers of this manual may despair. 
They may feel the task of control is so extensive 
and so difficult, and their available resources 
so limited, that the task is hopeless. There may 
be other readers who feel that the author is 
overly pessimistic about performance of survey 
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personnel, and over cautious; that many of the 
recommended control procedures are un- 
necessary, or even silly. To the first of these 
readers, we admit that the sources of possible 
error are many and diverse, and some of the 
control procedures are difficult to carry out. 
But be encouraged. Once instituted, most of 
the control techniques are less complex opera- 
tions than they appear at first. To those who 
think the proposed controls are over ambitious, 
we observe that every one of the hazards dis- 
cussed does exist, and suggest that a program 
of too much control is less dangerous than one 
of too little control. 

To all readers it is emphasized that the pri- 
mary message is to encourage the adoption of 
a policy of control, a habit of thinking that 
recognizes risk of error in any survey proce- 
dure, and seeks means of keeping that risk 
within tolerable bounds. It is not expected that 
every control procedure proposed will be intro- 
duced into every survey. Indeed it would be 
less than honest to convey the impression that 
most of the procedures discussed are present 
in most surveys. Some surveys must be con- 
ducted under unfavorable conditions, and with 
far from ideal personnel and resources. Never- 
theless, one of the most serious flaws in survey 
work is the discrepancy between intention and 
execution. Controls designed to narrow that 
discrepancy are among the most profitable in- 
vestments that a survey manager can make. 


Appendix A. A Brief Note on Acceptance 
Sampling and Process Control 


A1. Foran account of the theory underlying 
acceptance sampling and process control, one 
should consult a textbook on the subject, such 
as reference [4] or [7]. But it may be useful 
to note here a few of the key concepts and a 
little of the terminology in common use. 

A2. Operating Characteristic (OC) Curve. 
This curve is the key description of an accep- 
tance sampling plan. It gives the probability of 
accepting, A, a lot, under a specified plan for 
any fraction defective, p. The equation for the 
Curve Is 


A~3 (2 yer 2) ial 


where rn is the sample size (no. of items test- 
ed), and 
d is the number of items in the sample 
found defective, 


cis the acceptance number, which 
means that if d<c, the lot is ac- 
cepted, and otherwise the lot is 
rejected. 


The OC equation is usually plotted as a con- 
tinuous Curve (see Figure A-1). The shaded 
area at the left of the curve shows the region 
of acceptance for high quality production — 
where the probability of acceptance is near 
unity. The shaded area at the right shows the 
region of acceptance for very low quality work 
—where the probability of acceptance ap- 
proaches zero. The shape of the OC curve is, 
of course, a function of all the parameters of 
the curve: p, n, and c. In particular, the OC 
Curve is steeper, and thus more discriminating, 
for larger sizes n. A high level of quality is 
assured when c is small compared with n, but 
costs increase as c is decreased for given n. 

It will be observed that A is given by the sum 
of the first (c + 1) terms of the binomial ex- 
pansion [p + (71 — p)]". For the smaller 
values of p (which are the most important, in 
many situations), the value of A can be approx- 
imated by 


“— 


A’ = 3e-™ (np) / di. 
a=o 


A3. Average Outgoing Quality Curve (AOQ). 
This curve gives the expected average outgoing 
percent defective, B, for the chosen plan, when 
the incoming fraction defective is p. The AOQ 
Curve has the equation 

B= pA 

where A is given by the OC curve, and typi- 

cally has a shape as in Figure A-2. 
It will be observed that if the actual incoming 
fraction is either very low or very high, the 
expected outgoing fraction defective is low — 
low because either p is itself low or because 
the probability of acceptance is low if p is high. 
The worst possible result is measured by the 
maximum value of the AOQ curve, called the 
average outgoing quality limit, or AOQL. 

A4. Choice of Acceptance Plans. Theory 
for construction of ideal acceptance plans is 
Outside the scope of this manual. An outline 
of one simple approach and approximating 
method for choosing the acceptance number c 
makes use of the concept of average quality 
level (AQL), and can be presented here. One 
selects a target acceptable error rate (per cent 
defective), P, as the AQL. Then for each of 
several likely sizes of sample observations, n, 
which will have been largely determined by data 


OC Curve 


p= Fraction Defective 


Figure A-1. Operating Characteristic Curve (OC) 
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collection or processing procedures, a series 
of calculations is made. Let D be a hypothetical 
number of defectives in n observations, where 
the probability of an observation being defec- 
tive is P, the chosen AQL. The probability of D 
being realized is 


pid) = (5 )Pha = Pr-". 


Values of p(D) are calculated for D = 0, 7, 2, 3 
gig and their cumulative sum obtained until 
that cumulative sum first exceeds 0.95. If that 
value of Dis (c + 17), then c is the acceptance 
number. The rationale is to make the odds 
19 to 1 that the lot will be accepted when the 
per cent defective is P. If the actual per cent 
defective is less than P, the probability of ac- 
ceptance is greater than 0.95. If the actual 
per cent defective is greater than P, there is a 
less than five per cent chance of acceptance. 
This probability can be calculated by the same 
technique for any other hypothetical value of P’ 
different from P, by substituting P’ for P in the 
formulas. A cumulative average observed sam- 
ple error rate should be computed, and if it 
exceeds P by much, the acceptance number 
should be lowered, or a more sophisticated 
analysis undertaken, turning to the basic prin- 
ciples stated in A2 or A3, or to textbooks on 
the subject. 


100 
Average 
Outgoing 
Percent 
Defective 
B 
0 


The final column of Tables 7-2 and 8-1 is 
concerned with ‘‘Consumers Risk.”’ The initial 
objective was to be 95% sure of accepting 
the lot if the actual error rate is P. The ‘‘con- 
sumer’’ wants to Know how large an error rate 
he may be faced with at the 10 per cent accep- 
tance point on the OC curve. This is the figure 
in the final columns of Tables 7-2 and 8-1. 
Knowing the sample size, n, it can be located 
approximately by trial and error calculations, 
or by scanning and interpolating in a table of 
Cumulative sum of terms in a binomial or Pois- 
son distribution. For example, forn = 40, and 
Cc = 4, one finds using the Poisson Distribution, 
that 0.20 is the value of P’ which yields 
A‘ = 0.10. That is 


4 


a [40(0.20)]" / d! = 0.10. 

(This same solution can be obtained from 
tabled data such as is found in Table 6, page 
569 of reference [7], where one sees for c = 4, 
and the Poisson summation equals 0.10, that 
nP*‘ = 8.0; and since n = 40, P’ must equal 
0.20.) This means that there is a 10 per cent 
chance of accepting an error rate of 20 per 
cent, if one is using an acceptance number of 
4 in a sample of 40 observations. 


AOQL 


p 1 


Figure A-2. Average Outgoing Quality Curve (AOQ) 
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Appendix B. An Illustrative Procedure 
for Guidance in Selecting and 
Training Interviewers 


B-1. Purpose of the Appendix. Section 7.4 
above includes description of a model intended 
to be helpful to survey managers in setting 
standards for initial training and selection: of 
interviewers. It was said that the model cannot 
in practice be followed precisely, but that it 
might serve as an indicator of how we should 
proceed. The present index offers an illustra- 
tion of a pattern of thinking that makes use of 
the model. 

The illustration may appear to exhibit a more 
precise and sophisticated solution to the prob- 
lem than is really attained. Although the method 
of solution borrows from sound statistical] theory, 
the conditions in any given application are such 
that conclusions must be considered ‘‘reason- 
able judgments”’ rather than demonstrably vali- 
dated rules. 

The general rationale of a suggested ap- 
proach will be outlined first, and then presented 
in somewhat greater illustrative detail. It is 
recognized that no interviewer will be perfect, 
and that even those interviewers who are ac- 
ceptably proficient will vary among themselves 
in actual performance. The problem is then to 
determine a reasonable level of average perfor- 
mance for the chosen group of interviewers, an 
acceptable range of deviation from this average 
for any interviewer who is to be retained, and 
consequently the error rate for individual inter- 
viewers above which the interviewer should be 
dismissed. An operating solution to the prob- 
lem is secured through mesh of a theoretical 
model, speculation about typical performance, 
and observations in the pretest. 

B-2. An Example. The arguments are most 
easily illustrated in terms of a specific hypo- 
thetical example. Perhaps the survey design 
contemplates for several key statistics a sam- 
pling relative standard error of 3 per cent, and 
has set as working targets measurement relative 
Standard error of 5 percent [/.e., Vm? = 0.0025}, 
and relative bias not to exceed 10 per cent. 
Speculation on the allocation of measurement 
variance and bias to the various stages of sur- 
vey operation is a formidable task for which 


neither theory nor available empirical evidence 
give much guidance. But it is plausible to argue 
that at least half the relative bias and a sub- 
Stantial majority of the measurement relvariance 
will have their source in the agent operations 
of interview, transcription, and editing-coding. 
To get at the dimension of the standards prob- 
lem, let us say that the data collection and 
Processing operations must have a relative 
bias of not more than 5 per cent, and measure- 
ment relvariance of 0.0016. We speculate 
further that 4 points of the S-point bias and 
0.0009 of the 0.0016 relvariance come from the 
interview. 

Even accepting these speculations, a proper 
course of action has not yet been determined, 
for there remain many specific arrangements 
which would meet the requirements. But useful 
guidelines are beginning to take shape. Let x 
represent ‘‘correct’’ performance in interview- 
ing. Then the expected value of x’, the average 
value of the pooled performance of selected 
trained interviewers must not deviate from x 
by more than 4 per cent of the x value. 

A variance of the x’, arising from variation in 
the performance of individual interviewers, can 
be allowed up to the amount of 0.0009x?. 

The distribution of individual interviewers by 
the size of their performance scores is likely 
to be somewhat skewed, but for the purposes 
of this exercise, it can be assumed to follow the 
Gaussian Normal Curve. If the field work calls 
for, say, 100 interviewers, this means that a 
population variance among interviewers of 
ox = 100 (0.0009x?) = 0.09x? can be tolerated. 


Thus a relative standard deviation of 30 per 
cent is permitted. 

Consequently, a panel of trained interviewers 
must be selected for which in relative terms the 
expected pooled performance is within 4 per 
cent of the target while variations among the 
interviewers is held to a coefficient of variation 
of 30 per cent. 

We still do not have an unique solution to 
setting standards for selection of individual 
interviewers, but the following plan is a reason- 
able one, and will provide at least moderate 
confidence that the interviewing force is cap- 
able of adequate performance. 
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1. Determine the direction of expected bias. 
Usually this will be a negative bias or under- 
count. That is Ex’ — x = 8 will be a negative 
number. 

2 The model leads, then, to a desired dis- 
tribution of trained interviewers similar to that 
depicted in Figure B-1. 

3. A key feature of this example is that not 
more than 15 per cent of interviewers should 
have ‘‘deficiency’’ scores of more than 34 per 
cent, nor more than about 15 per cent an “eX- 
cess” score of more than 26 per cent. 

4. A slightly conservative interpretation 
(appropriate because necessary assumptions 
have been weakly based) argues that no inter- 
viewer should be accepted with a deficiency 
error as large as 65 per cent or an eXCess 
error as large as 55 per cent. 

5. The summary guideline then for select- 
ing, training, and rejecting interviewers uses, 
in this illustration, the following target pattern of 
test scores. 


Percent of “Correct” 


Criterion 
Measure x 


Not more than 4 
percent 


Average pooled error for all 
interviewers 


Not more than 15 inter- 
viewers with a ‘‘deficiency 
error greater than 


en a ee 


Not more than 15 inter- 
viewers with an ‘‘excess’’ 
error greater than 


34% 


26% 


Reject any interviewer with 
a ‘‘deficiency’’ error greater 
than 


65% 


Reject any interviewer with 
an ‘‘excess”’ error greater 
than 


5a Yo 


lt should be obvious that the argument of this 
appendix leads only to rough measures which 
can serve as guidelines for setting reasonable 
standards for interviewer performance. The 
objective has been to determine the general 


Number of 
Interviewers 


Value of x for sth Interview 


Figure B-1. Expected Performance of Individual Interviewers 
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dimensions of a rational course. Using a simi- 
lar pattern of reasoning, and taking account of 
the peculiar conditions of his survey, the Oper- 
ating statistician can adopt variations of this 
plan, which should lead to a sensible pattern 
of training, retraining, rejection of some 
trainees, and selection of the panel of inter- 
viewers. 
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